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DSMC  PREFACE 
WE  NEED  YOUR  HELP 


In  1986,  the  General  Accounting  Office  issued  a  report  on  then  current  DOD  efforts 
relative  to  technical  risk  assessment.  This  stimulated  many  introspective  activities  to  im¬ 
prove  the  management  of  risk  in  a  weapon  systems  program.  Updating  the  1983  DSMC 
Risk  Assessment  Techniques  guide  was  one  of  those  efforts.  This  updated  Guide  is  the  result. 

Introspective  activities  centered  on  three  key  points: 

Program  management  is  risk  management  or,  in  other  words,  the  program  manager's 
job  is  to  manage  risk. 

Management  of  risk  requires  not  only  the  management  of  technical  risk,  but  includes 
managing  cost  risk,  schedule  risk,  programmatic  risk  and  supportability  risk.  All  are  im¬ 
portant  in  program  management. 

There  are  no  "textbook"  answers  to  risk  management.  Each  situation  is  different  and 
each  circumstance  requires  a  slightly  different  approach.  Therefore,  it  should  be  obvious 
that  this  Guide  cannot  be  a  panacea.  It  presents  several  concepts  and  methodologies,  many 
from  the  acquisition  community,  which  have  been  integrated  into  a  holistic  framework. 
This  is  to  say  that  we  authors  may  have  missed  something  good  that  is  "out  in  the  field." 

We  solicit  your  help.  Use  this  Guide  for  awhile.  Try  it  out.  If  you  have  additional  infor¬ 
mation,  if  you  are  aware  of  other  methodologies,  if  you  used  other  techniques  or  approaches 
that  worked  for  you,  please  let  us  know.  The  Defense  Systems  Management  College  in¬ 
tends  to  revise  this  Guide  and  republish  it  in  FY90.  Please  send  your  comments  by  September 
30,  1989. 

The  Risk  Management  Guide  was  developed  by  The  Analytic  Sciences  Corporation  (TASC) 
under  contract  to  DSMC.  As  coordinator,  I  exttnd  my  thanks  to  all  offices  and  contractors 
providing  information.  I  want  to  acknowledge  the  extensive  efforts  and  valuable  inputs 
of  Mr.  Bernie  Rudwick  and  Commander  Tom  Withers  of  DSMC,  and  Mr.  Troy  Caver, 
formerly  of  DSMC,  whose  valuable  insight  and  assistance  contributed  materially  to  develop¬ 
ing  this  Guide. 


Harold  J.  Schutt 
Associate  Dean, 
Department  of  Research 
and  Information 
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FOREWORD 


This  Risk  Management  Guide  is  de¬ 
signed  to  provide  program  management  offices 
with  a  reference  book  for  dealing  with  program 
risk  related  to  systems  acquisition.  There  is  no 
single  ,,best"  technique  for  managing  risk. 
Thus,  the  guide  provides  ar-  introduction  to  the 
concepts  of  risk  and  an  overview  of  the  tech¬ 
niques  for  managing  risk.  This  approach  allows 
the  reader  to  select  the  most  appropriate  risk 
strategy  for  their  circumstances.  The  guide 
alerts  readers  to  the  many  problems  and  issues 
faced  in  acquisition  risk  management  and  deals 
with  many  of  the  issues  raised  in  the  1986  Gen¬ 
eral  Accounting  Office  (GAO)  report  on  Tech¬ 
nical  Risk  Assessments.  (Ref.  FI) 

This  guide  has  been  designed  to  be  used 
both  as  an  aid  in  classroom  instruction  and  as 
a  reference  book  for  practical  application.  It 
is  intended  to  aid  all  levels  of  program  manag¬ 
ers  and  designated  "risk’’  analysts.  More  expe¬ 
rienced  program  managers  may  want  to  skip 
the  introductory  chapter  and  some  of  the  ap¬ 
pendices.  The  chapter/reference  matrix  in  Fig¬ 


ure  FI  is  intended  to  serve  as  a  quick  look-  up 
chart  to  the  book  for  easy  field  reference. 

SCOPE 

This  handbook  is  limited  to  program/ 
project  risk  management  as  it  relates  to  the 
DOD  acquisition  process.  It  does  not  cover 
“insurance  risk”,  “safety  risk”,  or  "accident 
risk”  which  are  generally  considered  to  be  out¬ 
side  of  the  DOD  acquisition  management 
realm.  Focus  is  placed  on  risk  management 
from  a  program  office  viewpoint.  Program 
management  offices  are  charged  with  the  re¬ 
sponsibility  of  making  decisions  which  inher¬ 
ently  have  an  element  of  uncertainty.  As  can  be 
seen,  there  is  no  clear  cut  distinction  between 
program  management  and  risk  management. 
Risk  management  is  an  integral  part  of  the 
program  management  function.  Risk  manage¬ 
ment  should  be  thought  of  as  a  program  man¬ 
agement  methodology  rather  than  an 
independent  function  distinct  from  other  pro¬ 
gram  management  functions. 
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Figure  FI  Quick  Look-Up  Reference  Chart 


Risk  Management  is  “a  method  of  man¬ 
aging  that  concentrates  on  identifying  and  con¬ 
trolling  the  areas  or  events  that  have  a  potential 
of  causing  unwanted  change.  ”  “...  it  is  no  more 
and  no  less  than  informed  management.”  (Ref. 
F2) 

APPROACH 

Risk  is  approached  in  this  handbook 
from  a  holistic  viewpoint.  That  is,  risk  is  ad¬ 
dressed  as  a  single  entity,  consisting  of  different 
facets  (technical,  programmatic,  suppor- 
tability,  cost,  and  schedule)  as  illustrated  in 
Figure  FII.  While  technical  issues  are  a  primary 
source  of  risk  and  figure  prominently  through¬ 
out  the  book,  they  must  also  be  balanced  with 
the  management  of  other  aspects  of  the  pro¬ 
gram  (other  risk  facets).  Therefore,  a  fair 
amount  of  time  is  spent  examining  each  of 
the  facets  of  risk  so  the  reader  can  obtain  an 
understanding  of  the  inter-relationships  that 
can  exist  between  the  facets. 
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Figure  H  The  Five  Facets  of  Risk 


NOTES  ON  GUIDE  USE 

While  using  this  handbook,  keep  in 
mind  that  risk  is  a  complex  concept  subject  to 
individual  perception.  Some  people  are  risk 
takers  and  some  people  are  risk  averse.  Hence, 
it  is  difficult  to  develop  a  universal  set  of  rules 
for  dealing  with  risk.  Guidance,  structure,  and 
sample  handling  techniques  are  contained  in 
this  guide  which  follow  sound  management 
practices.  While  the  principles,  practices,  and 
theories  presented  herein  hold  true  in  nearly 
all  situations,  under  certain  circumstances  the 
rules  by  which  risk  is  evaluated  may  change 
drastically.  For  example,  in  times  of  extreme 
threat  people  do  extraordinary  things.  They 
will  take  risks  that  under  ordinary  circum¬ 
stances  would  be  deemed  “unacceptable". 
Hence,  high  risk  programs  are  not  always  bad, 
and  the  acquisition  of  high  risk  programs 
should  not  necessarily  be  avoided.  Rather,  they 
should  be  rigorously  monitored  and  con¬ 
trolled. 

In  developing  this  guide,  extensive  sur¬ 
veys  were  carried  out  with  over  70  program  of¬ 
fices  and  25  contractors*.  The  risk  techniques 
resulting  from  this  survey  effort  have  not  been 
evaluated  for  all  circumstances.  The  user  is  re¬ 
sponsible  for  determining  the  validity  and  ap¬ 
propriateness  of  a  particular  technique  for 
his/her  own  application. 

*  Over  380  surveys  were  actu¬ 

ally  sent  to  government  and 
industry. 
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Chapter  1 
INTRODUCTION 


“The  job  of  a  program  manager  is  to  allocate 
resources  to  achieve  goals  with  minimum  risk.” 


1.1  EXECUTIVE  SUMMARY  AND 

GUIDE  OVERVIEW 

The  risk  guide  is  structured  in  a  tutorial 
fashion.  It  begins  with  a  brief  review  of  the  his¬ 
tory  or  risk  management  within  DOD  and 
some  discussion  why  it  is  necessary  (Chapter 
2).  The  next  chapter  (Chapter  3)  then  defines 
risk  in  terms  relevant  to  program  management 
and  establishes  the  basic  concepts  necessary  to 
understand  the  nature  of  risk.  After  giving  the 
reader  an  understanding  of  the  basic  concepts 
of  risk,  the  book  then  defines  the  structure  and 
process  used  in  risk  management  which  can  be 
applied  to  all  program  phases.  Chapter  5  pre¬ 
sents  the  specific  techniques  necessary  to 
successfully  execute  the  process  set  out  in 
Chapter  4.  At  this  point,  readers  will  be  pre¬ 
pared  to  precede  with  the  actual  execution  of 
risk  management.  The  core  chapter  (Chapter 
6)  wraps  up  the  information  in  the  previous 
chapters  by  discussing  the  implementation  is¬ 
sues  program  managers  will  face  in  executing  a 


risk  management  program.  Two  special  topics 
are  then  covered  which  serve  as  supplemental 
material  to  the  process;  contractor  risk  man¬ 
agement  and  the  future  of  risk  management. 
This  “building  block”  order  is  represented  by 
Figure  1.1-1. 


The  risk  guide  also  contains  seven  ap¬ 
pendices  intended  to  serve  as  reference 
material  and  provide  backup  detail  for  some 
of  the  concepts  presented  in  the  text  of  the 
guide.  These  are  as  follows: 

•  Risk  Sources  -  an  abbrevi¬ 
ated  list  -  intended  to  be  a 
starting  risk  “checklist" 

•  Bibliography  -  this  is  a 
comprehensive  bibliogra¬ 
phy  on  acquisition  risk 
management 

•  Existing  Policy -this  is  a  se¬ 
ries  of  extracts  from  current 
policies  governing  the  ac¬ 
quisition  process  as  they 
relate  to  risk  management 
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BACKGROUND 


Figure  1.1-1  Risk  Guide  Structured  Approach 


•  Definitions/Acronyms 
self-  explanatory 

•  Basic  Probability  Concepts  - 
intended  as  a  refresher  and 
basic  primer  for  the  material 
in  the  text 

•  Quantifying  Expert  Judg¬ 
ments  -  narrative  of  ways  to 
transform  qualitative  infor¬ 
mation  into  quantitative 
information  during  expert  in¬ 
terviews 

•  Special  Notes  on  Software  - 
because  of  the  growing  com¬ 
plexity  and  difficulty  in 
managing  software  pro¬ 
grams,  this  appendix  was 
included  to  provide  a  starting 
point  for  help. 


1.2  NOTES  ON  THE  GAO  REPORT 

Within  the  1986  GAO  report  cited  ear¬ 
lier  (Ref  FI),  there  were  five  criteria  developed 
that  were  considered  essential  in  the  assess¬ 
ment  of  technical  risk.  These  criteria  really 
apply  to  more  than  just  “technical”  risk. 


(1)  prospective  assessment:  Possible 
future  technical  problems  are  considered,  not 
just  current  problems. 

(2)  planned  procedures:  Assessment  is 
planned  and  systematic,  not  incidental. 

(3)  attention  to  technical  risk:  There  is 
explicit  attention  to  technical  risk,  not  just  to 
schedule  or  cost  risk  with  consideration  of 
technical  risk  left  implicit. 

(4)  documentation:  At  a  minimum, 
technical  risk  assessment  procedures  and  re¬ 
sults  are  written  down  in  some  form. 

(5)  reassessment  in  each  acquisition 
phase:  New  or  updated  assessments  are  made 
in  order  to  detect  changes  in  risk  during  a  sys- 
tem’s  development. 

As  the  reader  progresses  through  the 
guide,  the  importance  of  adhering  to  these  cri¬ 
teria  will  become  evident.  Without  an 
understanding  of  the  complexity  of  dealing 
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with  risk,  these  criteria  appear  to  be  merely 
reasonable.  With  an  understanding  of  the  com¬ 
plexity  of  risk,  their  importance  is  seen  as 
critical.  It  is  the  intent  of  this  b'*  jk  to  bring  the 
reader  up  to  a  knowledge  level  where  these  cri¬ 
teria  are  viewed  as  mandatory  for  successful 
risk  management. 


i  NOTE.  The  GAO  used  the  term  “as¬ 
sessment”  to  include  a  more  comprehensive 
set  of  risk  activities  than  as  defined  in  this 
book.  (See  Chapter  4). 


1-3 


Chapter  2 
BACKGROUND 


2.1  HISTORY 

It  has  long  been  recognized  that  risk 
management  provides  valuable  information 
to  program  management  personnel.  Deputy 
Secretary  of  Defense  David  Packard  wrote  a 
memorandum  to  the  military  services  in  1969 
that  listed  inadequate  risk  assessment  as  a  ma¬ 
jor  problem  area  in  system  acquisition.  In 
1981,  Deputy  Secretary  of  Defense  Frank  C. 
Carlucci,  III,  published  a  memorandum  (Ref¬ 
erence  2-1)  which  included  32  “initiatives” 
(these  became  known  as  the  Carlucci  initia¬ 
tives)  aimed  at  improving  the  acquisition  proc¬ 
ess.  Initiative  11  required  Department  of 
Defense  (DOD)  action  to  increase  the  visibility 
of  technical  risk  in  budgets  of  weapon  systems 
acquisition  programs.  Then  in  1986,  the  United 
States  General  Accounting  Office  released  a 
report  titled;  Technical  Risk  Assessment  -  The 
Status  of  Current  DOD  Efforts ,  which  exam¬ 
ined  the  methodology  used  for  assessing  tech¬ 
nical  risks  within  25  program  offices 
(Reference  2-2).  The  deficiencies  found  by 


the  GAO  prompted  the  development  of  this 
guide. 

2.2  THE  ISSUE  OF  FORMALITY 

In  order  for  the  risk  management  proc¬ 
ess  to  work,  it  must  become  formal,  systematic, 
and  applied  in  a  disciplined  manner.  That  is 
not  to  say  that  all  programs  should  require  for¬ 
mal  risk  management.  It  merely  means  that  to 
obtain  the  maximum  benefit  from  risk  manage¬ 
ment,  it  must  become  a  systematic  process. 
There  have  been,  in  the  past,  several  problems 
which  prohibited  risk  management  from  be¬ 
coming  a  clearly  understood  process.  The  in¬ 
tent  of  this  book  is  to  address  these  problems 
and  thereby  lay  the  ground  work  for  institution¬ 
alizing  risk  management.  The  risk  manage¬ 
ment  “structure”  used  throughout  this  book  is 
depicted  in  Figure  2.2-1. 

This  structure  is  defined  in  Chapter  4. 
Note  that  “risk  management”  refers  to  the 
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Figure  2.2-1  Risk  Management  Structure 


sum  total  of  four  specific  elements.  Assess¬ 
ment,  Analysis,  and  Handling  refer  to  the  ac¬ 
tual  execution  of  the  process  while  Planning 
represents  most  of  the  preparation  activity. 

23  THE  RISK  MANAGEMENT  NEED 

Most  decisions,  including  the  most  sim¬ 
ple,  involve  risk.  Take,  for  example,  the  deci¬ 
sion  of  whether  to  drive  or  fly  on  a  business 
trip;  the  cost  and  time  differentials  are  easily 
obtained,  but  the  safety  factor  and  the  prob¬ 
ability  of  arriving  on  time  for  a  meeting  can 
become  very  complicated. 

With  this  example  in  mind,  a  “success 
criteria”  is  necessary  early  in  the  effort  in  or¬ 
der  to  set  down  the  most  important  elements 
in  the  risk  assessment.  If  cost  alone  is  the  only 
success  criterion,  then  the  risk  determination  is 
simple;  determine  the  cost  to  fly  and  compare 
this  to  the  expense  of  driving.  The  next  success 
criterion  might  be  safety.  One  method  of 


transportation  will  be  safer  than  the  other.  Sta¬ 
tistics  concerning  accidents  per  1000  miles 
traveled  are  available  to  evaluate  this  crite¬ 
rion.  If  a  third  criterion  is  added,  such  as  on- 
time  arrival  for  a  meeting,  then  dependability 
of  the  transportation  method  must  be  entered 
into  the  calculation.  Airline  on-time  statistics 
and  the  dependability  of  the  auto  and  the  road 
conditions  should  be  evaluated. 

As  the  success  criterion  is  expanded 
and  made  more  complicated,  the  decision¬ 
making  becomes  more  complicated.  It  is  obvi¬ 
ous  from  the  example  that  some  risk  (perhaps 
increased  cost)  is  acceptable,  while  being  late 
for  the  meeting  may  be  an  unacceptable  risk. 
Certainly,  not  arriving  safe  and  sound  is  com¬ 
pletely  unacceptable. 

Today’s  weapon  systems  are  increasing 
in  technical  complexity,  and  this  increasing 
complexity  increases  the  risk.  Program  deci¬ 
sions  are  heavily  biased  toward  cost  anti 
schedule  goals.  While  cost  and  schedule  are 
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understood,  the  impact  of  cost/schedule  deci¬ 
sions  as  they  relate  to  technical  performance 
risk  are  usually  not  as  clear.  A  formal  method¬ 
ology  for  evaluating  the  impacts  of  decision¬ 
making  and  foreseeable  problems  is  necessary. 
In  addition,  this  methodology  should  aid  in 
identifying  any  practical  and  effective  work¬ 
arounds  in  order  to  achieve  the  program  goals. 

Proper  risk  management  requires  a 
systematic  approach  to  the  identification  of 
problems.  The  sizing  and  resolution  of  these 
problems  can  only  help  in  the  determination 
of  choices,  given  certain  causes  and  effects.  In 
order  to  insure  that  the  approach  is  systematic, 
it  would  include  the  communication  of  risk  as 
seen  by  each  diverse  technical  function  to  the 
single  decision  maker  in  order  to  obtain  the 
maximum  program  benefit  in  terms  of  per¬ 
formance,  cost,  and  schedule. 

While  many  program  managers  use  in¬ 
tuitive  reasoning  (guessing)  as  the  starting 
point  in  the  decision-making  process,  it  be¬ 
hooves  the  astute  manager  to  go  beyond  the  in¬ 
tuitive  reasoning  or  experience  factor  in 
decisions  involving  significant  risks.  As  a  mini¬ 
mum,  a  manager  should  attempt  to  obtain  the 
level  of  risk  and  the  impact  of  the  action  on 
the  progress  of  the  program.  If  the  risk  is  of 
such  consequence  as  to  cause  the  entire  pro¬ 
gram  to  fail,  then  it  may  not  be  acceptable  and 
some  other  plan  must  be  formulated. 

In  today’s  defense  environment,  there 
are  factors  that  must  be  carefully  examined  for 
risk  in  order  to  understand  the  necessity  for 
risk  management.  The  project  manager  must 


be  aware  of  potential  cost  and  schedule  per¬ 
turbations;  frequently  the  survival  of  a  project 
(and  perhaps  the  manager)  depends  on  the 
control  of  these  elements. 

Given  the  above  description  of  the  de¬ 
fense  environment  and  the  qualifications  for 
effective  program  management,  it  is  advisable 
for  all  programs  to  perform  some  documented 
risk  management  activity,  either  qualitative  or 
quantitative.  All  DAB  programs  should  have 
formal,  intense  risk  management  activities 
while  smaller,  less  critical  programs  may  re¬ 
quire  less  effort.  The  ultimate  authority  is  the 
program  manager.  He  must  make  the  judg¬ 
ment  based  on  performance,  cost,  and  sched¬ 
ule  challenges  faced  on  the  project. 

2.4  CHAPTER  2  KEY  POINTS 


•  Risk  Management  is  re¬ 
quired  by  policy  (see  Ap¬ 
pendix  C) 

•  Risk  Management  should 
be  formal  and  systematic 

•  Risk  is  an  integral  part  of 
decision-making 

•  Greater  pressure  on  DoD 
requires  more  effective  risk 
management 

•  Most  programs  should  have 
some  level  of  documented 
risk  management  activity. 


References 

2-1  Carlucci,  F.C.,  III,  “Improving  the  Ac¬ 
quisition  Process,”  Memorandum  for  Secretar- 


2-3 


ies  for  the  Military  Departments,  Chairman  of 
the  Joint  Chiefs  of  Staff,  Under  Secretaries  of 
Defense,  Assistant  Secretaries  of  Defense, 
General  Counsel,  Assistants  to  the  Secretary  of 
Defense,  the  Deputy  Secretary  of  Defense, 
Washington,  D.C.,  April  30  1981. 

2-2  “Technical  Risk  Assessment:  The  Status 
of  Current  DoD  Efforts,”  General  Accounting 
Office,  April  1986. 


Chapter  3 
RISK  CONCEPTS 


3.1  EXPANDED  DEFINITION  OF  RISK 

Risk  is  defined  as  the  probability  of  an 
undesirable  event  occurring  and  the  signifi¬ 
cance  of  the  consequence  of  the  occurrence. 
This  is  different  than  uncertainty  which  con¬ 
siders  only  the  likelihood  of  occurrence  of  the 
event.  (The  traditional  view  of  risk  defines  it  as 
a  situation  in  which  an  outcome  is  subject  to  an 
uncontrollable  random  event  stemming  from  a 
known  probability  distribution.  Uncertainty  is 
normally  thought  of  in  traditional  terms  as  an 
outcome  subject  to  an  uncontrollable  random 
event  stemming  from  an  unknown  probability 
distribution.  While  these  definitions  have  their 
place  in  statistics,  they  are  of  limited  value  in 
program  or  project  risk  management.)  Al¬ 
though  risk  and  uncertainty  are  often  used  in¬ 
terchangeably,  they  are  not  the  same.  What 
this  means  to  the  program  management  office 
is  that  to  truly  understand  whether  an  item  is 
“risky”,  they  must  have  an  understanding  of 
the  potential  impacts  resulting  from  the  oc¬ 
currence/nonoccurrence  of  the  event.  Figure 
3.1-1  illustrates  this  concept.  Note  Mat  some 


judgment  must  be  used  in  determining  risk  in 
this  manner.  For  example,  an  event  may  have 
a  low  likelihood  of  occurring,  but  the  conse¬ 
quences  of  the  event,  should  it  occur,  can  be 
catastrophic.  Most  people  would  not  consider 
this  to  be  a  high  risk  item  as  might  be  indicated 
in  the  Figure  3.1-1  diagram.  This  situation 
can  be  related  to  that  of  flying  in  a  commer¬ 
cial  jet  aircraft.  The  probability  of  a  crash  is 
low,  but  generally,  the  consequences  are  in¬ 
deed  grave.  While  most  people  do  not  con¬ 
sider  flying  a  high  risk,  many  do  feel  uncom¬ 
fortable  because  of  the  consequences  of 
“failure”.  This  example  also  highlights  the 
great  degree  of  subjectiveness  in  actually  rat¬ 
ing  risk.  It  is  highly  dependent  on  an  individu¬ 
al’s  perception  of  what  is  personally  accept¬ 
able.  Using  Figure  3.1-1  as  a  reference,  there 
are  three  separate  inputs  required  to  deter¬ 
mine  the  level  of  risk.  The  first  input  is  the 
“probability  of  occurrence  of  the  event.”  This 
variable  can  often  be  estimated  using  statisti¬ 
cal  references  based  on  history.  Probability 
theory  can  play  an  important  role  in  determin¬ 
ing  the  value  of  this  variable. 
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Figure  3.1-1  Concept  of  Risk 

The  second  input  is  “severity  of  conse¬ 
quence  if  the  event  should  occur".  This  vari¬ 
able  requires  the  management  team  to  identify 
what  the  consequences  are  and  the  degree  of 
the  impact.  Here,  statistics  and  probability  the¬ 
ory  can  play  a  role  in  determining  the  degree  of 
impact  once  it  has  been  identified.  Note,  how¬ 
ever,  that  probability  has  a  limited  role  and  is 
not  always  appropriate. 

The  third  input  required  is  subjective 
judgment  concerning  the  combination  of  the 
first  two.  There  can  be  little  disagreement 
about  the  level  of  risk  if  the  first  two  variables 
are: 

•  low  likelihood/low  conse¬ 
quence  -  low  risk 

•  high  likelihood/high  conse¬ 
quence  -  high  risk 


•  high  likelihood/low  conse¬ 
quence  -  low  risk  (to  the 
overall  success  of  the  pro¬ 
gram). 

As  you  move  towards  the  low  likeli¬ 
hood/high  consequence  quadrant  of  the  fig¬ 
ure,  the  risk  level  becomes  more  subject  to  in¬ 
dividual  interpretation  and  requires  strict 
program  guidelines  for  rating  the  risk.  Dis¬ 
agreements  among  participants  may  occur  in 
rating  risk.  While  program  managers  must 
rely  on  several  “technical  experts"  in  the  risk 
management  process,  they  must  also  be  pre¬ 
pared  to  make  the  final  judgment  on  the  rat¬ 
ing  of  risk.  Some  guidelines  on  the  rating  of 
risk  are  contained  in  4.3-2  of  this  guide.  It  is 
important  to  note  that  a  program  with  many 
moderate  risk  items  may  in  fact  be  a  high  risk 
program,  while  a  program  with  just  a  few  high 
risk  items  may  have  a  lower  overall  risk  rating. 
These  situations  usually  require  some  type  of 
modeling  to  ascertain  the  “program"  risk  level. 

Many  attempts  have  been  made  to 
mathematically  model  this  subjective  quantifi¬ 
cation  of  risk.  Probability  distributions  are  one 
such  method  frequently  used  (see  Appendix 
E).  One  last  item  to  be  considered  in  looking 
at  the  nature  of  risk  is  the  concept  of  opportu¬ 
nity.  There  must  always  be  some  potential  gain 
from  successfully  executing  an  activity  with 
risk.  As  the  potential  gain  increases,  so  does 
the  acceptability  of  higher  levels  of  risk.  If 
there  is  no  real  opportunity,  then  there  is  no 
reason  to  pursue  an  activity  with  risk. 
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3.2  RISK  FACETS 

After  obtaining  an  understanding  of  the 
nature  of  risk,  the  next  step  is  to  lay  the  ground¬ 
work  for  managing  it.  Risk  must  be  segmented 
into  manageable  pieces.  The  first  “cut”  is  to 
break  it  into  classifications  relating  to  the 
source  of  the  risk. 

3.2.1  Introduction 

Risks  to  a  program  manager  are  all 
rooted  in  the  determination  to  deliver  a 
specified  product  or  level  of  performance  at  a 
specified  time  for  a  specified  cost.  The  pro¬ 
gram  manager  risks  failure  in  three  ways  and 
combinations  thereof.  The  product  may  not  be 
up  to  the  performance  level  specified,  the  ac-  ■ 
tual  costs  may  be  too  high,  or  delivery  may  be 
too  late.  A  wide  variety  of  problems  can  arise 
to  keep  a  program  manager  from  meeting 
cost,  schedule,  and  performance  objectives. 
All  programs  that  are  properly  planned  will 
provide  the  manager  with  some  reserve  funds 
and  slack  time  to  work  around  unanticipated 
problems  and  still  meet  original  cost,  schedule, 
and  performance  goals.  There  is,  of  course,  a 
risk  that  the  original  cost,  schedule,  and  per¬ 
formance  goals  were  unattainable,  unrealistic, 
or  conflicting  and  it  would  be  impossible  to 
meet  all  of  them. 

There  are  five  facets  of  risk  that  are  nec¬ 
essary  to  segment  and  manage  the  cost,  sched¬ 
ule,  and  performance  issues  faced  on  a  project: 

•  Technical  - 

(performance  related) 


•  Supportability  - 
(performance  related) 

•  Programmatic  - 
(environment  related) 

•  Cost 

•  Schedule. 

Cost  and  schedule  risks  are  treated 
somewhat  differently  than  the  other  three  in 
that  they  are  (more  or  less)  indicators  of  pro¬ 
ject  status.  Note,  however,  that  cost  and 
schedule  can  become  a  major  source  of  pro¬ 
gram  risk.  This  will  be  discussed  in  detail  later. 

3.2.2  Classifying  Risk  into  the 
Facets 

Understanding  and  classifying  a  risk 
into  one  or  more  of  the  five  facets  requires  an 
examination  of  the  source  of  the  risk.  It  is  not 
always  easy  to  determine  intowhi.h  category 
a  particular  risk  belongs,  and  just  for  the  sake 
of  classification,  it’s  not  all  that  important. 
However,  understanding  the  source  of  the  risk 
and  the  impact  area(s)  as  well  as  providing  a 
structure  to  examine  risk  are  critical  elements 
if  the  risk  is  to  be  managed  effectively.  Figure 
3.2-1  depicts  sample  risks  from  each  facet. 

3.2.3  Technical  Risk 

Technical  risk  can  be  defined  as  the  risk 
associated  with  evolving  a  new  design  to  pro¬ 
vide  a  greater  level  of  performance  than  previ¬ 
ously  demonstrated,  or  the  same  or  a  lesser 
level  of  performance  subject  to  some  new  con¬ 
straints.  The  nature  and  causes  of  technical 
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Figure  3.2-1  Sample  Risks  by  Facet 


risks  are  as  varied  as  military  system  designs. 
Many,  if  not  most,  technical  risks  are  the  result 
of  the  demand  for  ever  greater  performance 
from  new  systems  and  equipment.  What  is 
technically  risky  when  first  attempted  may  be 
routine  a  few  years  later.  Risky  areas  on  a  sys¬ 
tem  with  high  performance  requirements  may 
be  routine  on  other  systems  with  lower  per¬ 
formance  requirements.  The  ever  present  re¬ 
quirement  to  minimize  or  maximize  physical 
properties  of  systems  and  equipment  further 
adds  to  the  risks  associated  with  higher  per¬ 
formance  requirements. 

Many  of  the  “ilities”  such  as  reliability, 
maintainability,  etc.  must  be  addressed  in  sys¬ 
tem  acquisition.  Each  can  be  viewed  as  addi¬ 
tional  design  requirements  placed  on  designers 
attempting  to  evolve  an  efficient  design  capa¬ 
ble  of  the  desired  performance  level.  Each  of 
these  added  design  requirements  can  be  a 
source  of  risk. 

It  is  not  easy  to  describe  all  possible 
technical  risks,  because  when  examined  at  the 
lowest  level  of  detail,  there  are  so  many  of 
them.  There  are  usually  many  items  to  be  de¬ 
signed  and  integrated  with  other  items.  There 
may  be  several  design  objectives  for  each  site 
and  each  item-design  objective  combination  is 
subject  to  many  “ility”  requirements,  as  well  as 
cost  and  schedule  constraints.  Appendix  A  con¬ 
tains  an  abbreviated  list  of  technical  risk  areas. 
It  does  not  break  out  types  of  risks  by  compo¬ 
nents,  parts,  subassemblies,  assemblies,  sub¬ 
systems,  and  systems  for  all  the  many  associ¬ 
ated  integration  design  tasks.  The  list  also  does 


not  address  all  possible  aspects  of  perform¬ 
ance,  which  vary  widely  from  system  to  system. 
As  the  design  architecture,  performance,  and 
other  requirements  and  program  constraints 
become  known  on  a  given  program,  a  more  de¬ 
tailed  list  of  risks  should  be  prepared  based  on 
system  peculiar  information. 

3.2.4  Programmatic  Risk 

Programmatic  risk  can  be  defined  as 
those  risks  which  include  obtaining  and  using 
applicable  resources  and  activities  which  may 
be  outside  of  the  program’s  control,  but  can  af¬ 
fect  the  program’s  direction.  Generally,  pro¬ 
grammatic  risks  are  not  directly  related  to  im¬ 
proving  the  state-of-the-art.  Programmatic 
risks  are  grouped  into  categories  based  on  the 
nature  and  source  of  factors  that  have  the  po¬ 
tential  to  disrupt  the  program  implementation 
plan. 

•  Disruptions  caused  by  deci¬ 
sions  made  at  higher  levels 
of  authority  directly  related 
to  the  program 

•  Disruptions  caused  by 
events  or  actions  affecting 
the  program,  but  not  di¬ 
rected  specifically  at  it 

•  Disruptions  caused  primar¬ 
ily  by  the  inability  to  fore¬ 
see  production  related 
problems. 

•  Disruptions  caused  by  im¬ 
perfect  capabilities 

•  Disruptions  caused  primar¬ 
ily  by  the  inability  to  foresee 
problems  other  than  those 
included  in  the  first  four 
categories. 
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These  risks  tend  to  be  a  function  of  the 
business  environment.  Appendix  A  has  a  more 
detailed  listing  of  sample  programmatic  risks. 

3.2.5  Supportability  Risk 

Supportability  risk  can  be  defined  as 
the  risk  associated  with  fielding  and  maintain¬ 
ing  systems  which  are  currently  being  devel¬ 
oped  or  have  been  developed  and  are  being 
deployed.  Note  that  supportability  risk  is  com¬ 
prised  of  both  technical  and  programmatic  as¬ 
pects.  Certainly,  any  design  effort  (which  may 
contain  technical  challenges)  should  consider 
what  the  supportability  issues  are  likely  to  be 
when  the  system  is  fielded.  Another  example  is 
training,  which  is  generally  a  programmatic 
risk  but  quickly  becomes  a  supportability  risk 
when  maintenance  and  operations  support  be¬ 
come  the  driving  factors.  There  are  ten  Inte¬ 
grated  Logistic  Support  Elements  that  present 
potential  sources  of  risk.  These  involve  both 
technical  and  programmatic  issues. 

•  Maintenance  Planning 

•  Manpower  &  Personnel 

•  Support  Equipment 

•  Technical  Data 

•  Training 

•  Training  Support 

•  Computer  Resources  Sup¬ 
port 

•  Facilities 

•  Packaging,  Handling,  Stor¬ 
age,  and  Transportation 

•  Design  Interface. 


It  is  important  to  understand  that  any 
given  risk  area  may  belong  to  more  than  one 
facet  as  illustrated  above  (e.g.,  a  particular 
piece  of  support  equipment  may  pose  a  tech¬ 
nical  challenge  and  have  significant  suppor¬ 
tability  implications). 

3.2.6  Cost  and  Schedule  Risk 

There  is  a  long  history  of  DoD  Weapon 
program  cost/schedule  growth  with  consider¬ 
able  Congressional  criticism  thereof.  In  an 
era  of  limited  DoD  budgets,  cost  and  schedule 
growth  in  one  program  dictates  reductions  in 
one  or  more  others.  Therefore,  the  risk  of  cost 
and  schedule  growth  is  a  major  concern.  This 
problem  is  further  complicated  by  the  fact  that 
performance  and  design  technical  problems 
are  sometimes  solved  by  increasing  the 
planned  program  scope  and  thereby  program 
cost  and  schedule. 

Cost  and  schedule  growth  is  the  differ¬ 
ence  between  the  estimated  program  cost  and 
schedule  and  the  actual  cost  and  schedule. 
Therefore,  there  are  two  major  cost/schedule 
risk  areas  bearing  on  cost/schedule  growth. 

•  The  risk  that  the  estimate 
set  an  unreasonably  low 
cost/schedule  objective 

•  The  risk  that  the  program 
will  not  be  carried  out  in  an 
efficient  and  prudent  man¬ 
ner  so  as  to  meet  reason¬ 
able  cost/schedule  objec¬ 
tives. 
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The  outcome  of  the  second  of  these  two 
risk  areas  is  not  primarily  a  cost/schedule 
analysis  related  risk,  that  is,  anything  cost/ 
schedule  analysts  or  financial  analysts  can 
control.  The  final  cost/schedules  are  primarily 
a  function  of  the  skill  of  the  Program  Manager 
to  accommodate  unanticipated  problems  re¬ 
lated  to  technical,  supportability,  and  program¬ 
matic  risks.  The  solution  or  the  lack  of  a  good 
solution  for  such  problems  often  increases 
costs  and  schedules. 

The  preparation  of  an  unrealistically 
low  baseline  cost/schedule  estimate  or  pro¬ 
gram  target  cost/schedule  estimate  fall  into 
four  categories  (prior  to  a  pricing  decision). 
These  are: 

•  Inadequate  system  descrip¬ 
tion 

•  Inadequate  historical  cost/ 
schedule  data  base 

•  Lack  of  sound  methods  re¬ 
lating  historical  costs/ 
schedules  to  new  program 
costs 

•  Incomplete  cost/schedule 
estimate. 

Note  that  from  this  context,  there  are 
few  true  cost  or  schedule  risks.  There  are  occa¬ 
sions  where  this  statement  does  not  hold  true. 
For  example,  test  windows  can  drive  entire 
programs  to  a  degree,  as  can  funds  available  for 
a  specific  item.  Generally,  true  cost  and  sched¬ 
ule  risks  are  few  and  far  between  when  the 
source  of  the  risk  is  closely  examined.  More 
often  than  not,  cost  and  schedule  uncertainty 
are  a  reflection  of  technical,  programmatic, 
and  supportability  risk. 


3.2.7  Facet  Organization 

It  was  mentioned  previously  that  there 
are  “risk  drivers”  and  “risk  indicators."  The 
risk  drivers  are  usually  the  technical,  program¬ 
matic,  and  supportability  facets  -  which  leave 
the  cost  and  schedule  facets  as  the  indicators. 
This  is  often,  but  not  always  the  case.  Gener¬ 
ally  when  an  item  is  contracted  for,  there  is  a 
specified  performance  level  to  be  met.  This  in¬ 
cludes  design  criteria,  supportability  factors, 
performance  criteria  and  a  host  of  other  specif¬ 
ics.  It  is  then  asked  what  it  will  actually  take  to 
build  this  item  in  terms  of  resources  (time  and 
money).  It  is  paramount  that  the  item  satisfy 
the  need.  The  tendency  then  is  to  focus  on  the 
performance  requirements  -  not  cost  or 
schedule.  Unfortunately  cost  and  schedule 
tend  to  be  the  yardstick  by  which  decisions  are 
made  -  and  the  tradeoffs  between  cost,  sched¬ 
ule,  and  performance  are  not  well  understood. 
This  is  one  of  the  advantages  of  performing 
risk  management.  It  attempts  to  draw  reality 
into  the  relationship  between  the  risk  facets. 
There  are  occasions  where  a  project  is  under¬ 
taken  with  the  understanding  that  the  product 
will  be  the  best  possible  within  the  dollar  and 
time  constraints  dictated.  In  these  instances 
the  cost  and  schedule  facets  become  the  driv¬ 
ers  and  the  other  facets  may  become  the  indi¬ 
cators.  Few  projects  have  such  clear  cut  goals. 
More  often  than  not,  the  program  manage¬ 
ment  office  must  strive  to  achieve  a  balance 
between  the  facets  to  reach  seemingly  conflict¬ 
ing  goals  in  performance,  cost,  and  schedule. 
For  simplicity,  this  guide  will  treat  technical 
risk,  programmatic  risk,  and  supportability 
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risk  as  the  predominate  factors  driving  cost 
risk  and  schedule  risk.  This  is  illustrated  in  Fig¬ 
ure  3.2-2. 


TECHNICAL 


Figure  3.2-2  Relationship  Between 
The  Five  Risk  Facets 

By  now  it  is  easy  to  see  that  the  risk  fac¬ 
ets  are  not  independent  of  one  another.  While 
a  design  risk  is  of  a  technical  nature  it  may  have 
cost,  schedule,  supportability,  and  program¬ 
matic  impacts.  Or,  a  tight  test  window  present¬ 
ing  a  schedule  risk,  may  have  serious  techni¬ 
cal  impacts.  The  facets  may  also  change  with 
time.  What  started  as  a  technical  risk  in  the  de¬ 
sign  of  a  product  may  surface  years  later  as  a 
supportability  risk  factor  that  has  serious  cost 
and  schedule  impacts.  A  useful  approach  is  to 
examine  all  facets  whenever  a  risk  is  identified 
in  one  facet. 

This  discussion  was  not  intended  to  im¬ 
ply  that  cost  and  schedule  manage  themselves; 


that  is  certainly  not  true.  The  intent  was  to  em¬ 
phasize  the  importance  of  managing  the  source 
of  the  risk  in  any  program.  Frequently,  this  is 
some  factor  rooted  in  technical,  programmatic, 
or  supportability  characteristics. 

33  OTHER  RELEVANT 

CONSIDERATIONS 

There  are  two  other  points  worthy  of 
mention  when  discussing  risk  concepts  from  a 
program  office  viewpoint.  Both  deal  with  our 
acquisition  management  structure  (to  a  de¬ 
gree)  and  are  discussed  in  the  following  two 
sections. 

3.3.1  The  Two  Perspectives  of 
Risk  Management 

Program/project  risk  management 
must  be  viewed  from  two  perspectives  defined 
as  follows: 

•  Short  term  -  dealing  with 
the  current  program  phase 
and  immediate  future 

•  Long  term  -  dealing  with 
anything  beyond  the  short 
term. 

Like  many  other  aspects  of  risk  man¬ 
agement,  the  distinction  between  the  two  per¬ 
spectives  is  somewhat  unclear.  Further  expla¬ 
nation  will  help  to  clarity  and  justify  the 
separation.  The  short  term  perspective  nor¬ 
mally  refers  to  managing  risk  related  to  satisfy¬ 
ing  the  immediate  needs  of  the  project  -  i.e., 
“this  is  the  performance  level  1  need  to  achieve 
today”,  and,  “how  are  my  contractors  manag- 
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ing  to  achieve  this?”  The  long  term  perspec¬ 
tive  deals  with  “what  can  I  do  today  to  ensure 
that  downstream  the  program  will  be  success¬ 
ful?”  This  might  include,  among  other  things, 
introducing  supportability  engineering  and 
producibility  engineering  into  the  design  proc¬ 
ess  early  in  the  program.  The  two  perspectives 
are  closely  related.  In  achieving  the  desired 
performance  level  (short  term  goal)  materials 
that  are  difficult  to  work  with  and/or  require 
new  manufacturing  techniques  as  yet  un¬ 
proven  may  be  utilized  to  solve  the  problem 
(introducing  a  long  term  risk).  As  with  any 
good  management  decisions,  the  shortterm 
and  long  term  implications  must  be  well  un¬ 
derstood.  Only  if  these  implications  are  known 
can  they  be  acted  on  (risk  handling)  early 
enough  to  significantly  reduce  the  chance  of 
undesirable  results.  Another  look  at  the  two 
perspectives  to  aid  in  understanding  the  differ¬ 
ences  is  illustrated  in  Figure  3.3-1.  In  this  fig¬ 
ure  an  overall  design  has  been  selected  for  a 
given  project  which  has  certain  elements  of 
risk.  This  was  a  decision  that  obviously  had 
long  term  implications.  The  task  now  at  hand 
for  the  program  manager  is  to  complete  the  de¬ 
sign  selected  within  the  resources  made  avail¬ 
able.  This  particular  program  manager  has  se¬ 
lected  some  technical,  cost,  and  schedule 
parameters  to  manage  “risk”  on  an  opera¬ 
tional  day  to  day  basis  (short  term  risk  manage¬ 
ment).  Again,  this  does  not  preclude  his  deci¬ 
sions  in  managing  short  term  risk  from  having 
significant  long  term  impacts. 


3.3.2  Realities  of  the  Government 
Program  Office  Function 

Under  ideal  program  management 
conditions,  the  same  management  team  would 
stay  with  a  program  from  the  definition  phase 
through  production.  However,  ideal  conditions 
rarely  exist  and  a  given  program  will  likely  see 
several  management  teams.  The  transition  in 
program  management  personnel  often  creates 
voids  in  the  risk  management  process  These 
voids  are  in  the  information/knowledge  gained 
about  the  program  from  previous  activity.  Pre¬ 
cious  time  must  be  spent  becoming  familiar 
with  the  program  —  often  at  the  sacrifice  of 
long  term  planning  and  risk  management.  The 
introduction  of  a  formal  system  for  recording, 
analyzing,  and  acting  on  program  risk  facili¬ 
tates  the  transition  process  and,  when  done 
properly,  forces  long  term  risk  management. 
The  approach  to  formal  risk  management  is 
contained  in  Chapters  4, 5,  and  6.  While  it  is  de¬ 
sirable  to  make  decisions  based  on  long  term 
implications,  it  is  not  always  feasible.  The  pro¬ 
gram  management  office  is  often  forced  to  act 
on  risk  from  a  short  term  rather  than  long  term 
perspective.  One  reason  has  already  been  men¬ 
tioned  -  the  change  in  personnel.  Another  rea¬ 
son  is  program  advocacy.  Sudden  shifts  in  pri¬ 
orities  can  wreak  havoc  on  long  term  plans 
(this  is  a  risk  area  in  and  of  itself).  The  result  is 
short  term  actions  to  adjust  to  the  new  priori¬ 
ties.  Often  these  kinds  of  decisions  are  made 
before  a  thorough  evaluation  of  the  long  term 
impacts  can  be  conducted.  Lastly,  in  some  in¬ 
stances  long  term  impacts  are  not  always  vis¬ 
ible  at  the  time  the  decision  must  be  made. 


3-9 


2 

•a 

o 

I 

it, 


fi 


00 


3 

■a 

u 


*■4 

n 

t 

& 

C 


3-10 


There  are  day  to  day  operational  risks 
that  must  be  addressed  to  complete  any  given 
phase  of  a  program.  The  solutions  developed 
to  handle  these  risks  must  always  be  exam¬ 
ined  from  a  long  term  viewpoint  and  must 
provide  the  program  manager  a  strong  argu¬ 


ment  to  defend  his/her  position.  As  has  been 
pointed  out  in  many  studies,  actions  taken 
early  in  a  program’s  development  have  a  ma¬ 
jor  effect  on  the  overall  performance  and  cost 
over  the  life  of  the  program  as  illustrated  in 
Figure  3.3-2.  (Ref.  3-1). 


i  i  in  years 
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Figure  3.3-2  Life  Cycle  Cost 
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3.4  CHAPTER  3  KEY  POINTS 


•  Risk  considers  both  likeli¬ 
hood  and  consequence 

•  Rating  risk  is  a  subjective 
process  requiring  strict 
guidelines 

•  There  are  five  facets  to  risk: 

-  technical 

-  progr  ammatic 

-  supportability 

-  cost 

-  schedule 

•  The  risk  facets  are  strongly 
interrelated 

•  Most  risk  sources  are 
rooted  in  technical,  pro¬ 
grammatic,  or  suppor¬ 
tability  factors 

•  Risk  has  ,a  short  term  and 
long  term  perspective. 
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Chapter  4 

THE  RISK  MANAGEMENT  STRUCTURE 


4.1  INTRODUCTION 


This  chapter  presents  the  recom¬ 
mended  structure  for  executing  risk  manage¬ 
ment.  Recognition  must  be  given  to  the  fact 
that  in  the  past  there  have  been  several  differ¬ 
ent  structures  and  definitions  used  for  basically 
the  same  concept.  This  has  been  a  source  of 
continuing  confusion  in  the  field  of  risk  man¬ 
agement.  Figure  4.1-1  illustrates  the  most 
common  of  the  previous  terminology/struc¬ 
tures  used  in  the  risk  field.  It  is  important  to 
note  that  all  of  these  previous  structures/ap¬ 
proaches  do  not  clearly  distinguish  between  the 
terms  risk  assessment/risk  analysis/risk  man¬ 
agement.  Previous  efforts  have  not  established 
standard  terminology.  This  chapter  will  clarify 
and  define  each  of  these  terms  so  that  commu¬ 
nications  regarding  “risk”  can  be  more  effec¬ 
tive.  Risk  management  consists  of  four  sepa¬ 
rate  but  related  activities  as  depicted  in  Figure 
4.1-2.  “Risk  Management”  is  the  “umbrella” 
title  for  the  processes  used  to  manage  risk.  This 


chapter  focuses  on  defining  and  explat  ning  the 
elements  of  risk  management.  As  with  any 
process,  there  are  two  basic  stages:  Planning, 
then  execution,  which  includes  monitoring  and 
control  (Reference  4-1). 


42  RISK  PLANNING 

4.2.1  Need/Purpose 

Risk  is  present  in  some  form  and  degree 
in  most  human  activity.  It  is  certainly  present 
in  the  systems  acquisition  business.  Risk  is 
characterized  by  the  fact  that: 

•  It  is  usually  at  least  partially 
unknown 

•  It  changes  with  time 

•  It  is  manageable  -  in  the 
sense  that  human  action 
may  be  applied  to  change 
its  form  and  degree. 

Planning  for  the  management  of  risk 
makes  ultimate  sense  in  order  to: 
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Figure  4.1-1  Previous  Risk  Structure 


Figure  4.1-2  Updated  Risk  Management  Structure 
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•  Eliminate  risk  wherever 
possible 

•  Isolate  and  minimize  risk 

•  Develop  alternate  courses 
of  action 

•  Establish  time  and  money 
reserves  to  cover  risks  that 
cannot  be  avoided. 


The  purpose  of  risk  management  plan¬ 
ning  is  simply  to  force  organized  purposeful 
thought  to  the  subject  of  eliminating,  minimiz¬ 
ing,  or  containing  the  effects  of  undesirable  oc¬ 
currences. 


4.2.2  Timing 


Risk  is  a  word  that  exists  only  in  the  fu¬ 
ture  tense.  There  are  no  past  risks  -  only  actual 
occurrences. 


Risk  management  planning  is  sensibly 
done  and  redone  as  an  integral  part  of  normal 
program  planning  and  management.  Some  of 
the  more  obvious  points  for  revisiting  the  risk 
management  plan  include: 


•  In  preparation  for  major 
decision  points 

•  In  preparation  for  and  im¬ 
mediately  following  techni¬ 
cal  reviews  and  audits  (see 
MIL-STD-1521) 

•  Concurrent  with  the  review 
and  update  of  other  pro¬ 
gram  plans  and  specifica¬ 
tions 


•  In  preparation  of  POM  sub¬ 

mittals. 

4.2.3  Risk  Management  Plan 

Most  major  programs  are  guided  by  a 
series  of  “plans”  (e.g.,  PMP,  TEMP)  that  pro¬ 
vide  the  rationale  and  intended  processes 
through  which  the  program  will  be  executed.  A 
Risk  Management  Plan  is  a  sensible  part  of  this 
suite  of  guiding  documents.  Such  a  plan  would 
publish  the  results  or  latest  status  of  the  risk 
management  planning  process. 

At  this  writing,  the  concept  of  a  Risk 
Management  Plan  is  gaining  favor  within 
DOD.  The  content  and  format  are  not  nearly  as 
mature  as  the  other  plans.  Thus  program  man¬ 
agers  have  almost  total  freedom  to  structure 
the  document  to  suit  their  situation.  As  a  start¬ 
ing  point,  consider  the  following  paragraphs  as 
a  guide  to  the  possible  content  of  a  Risk  Man¬ 
agement  Plan  (Figure  4.2-1). 

System  Description  and  Program  Sum - 
mary-  This  material  should  be  the  same  in  all 
the  program’s  plans.  It  should  provide  the  basis 
of  reference  for  the  reader  to  understand  the 
operational  need,  the  mission,  and  the  major 
functions  of  the  system.  It  should  include  key 
operational  and  technical  characteristics  of  the 
system.  A  program  summary  would  include  a 
description  of  the  organizational  relationships 
and  responsibilities  of  the  participating  organi¬ 
zations.  It  would  also  include  an  integrated 
program  schedule. 
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[7  PART  I  -  DESCRIPTION 

1.1  MISSION 

1.2  SYSTEM 

1.2.1  SYSTEM  DESCRIPTION 

1.2.2  KEY  FUNCTIONS 

1 .3  REQUIRED  OPERATIONAL  CHARACTERISTICS 

1 .4  REQUIRED  TECHNICAL  CHARACTERISTICS 

2.  PART  II  -  PROGRAM  SUMMARY 

2.1  SUMMARY  REQUIREMENTS 

2.2  MANAGEMENT 

2.3  INTEGRATED  SCHEDULE 

3.  PART  III  -  APPROACH  TO  RISK  MANAGEMENT 

3.1  DEFINITIONS 

3.1.1  TECHNICAL  RISK 

3.1.2  PROGRAMMATIC  RISK 

3.1.3  SUPPORTABILITY  RISK 

3.1.4  COST  RISK 

3.1.5  SCHEDULE  RISK 

3.2  STRUCTURE 
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Figure  4.2-1  Risk  Management  Plan 


Approach  to  Risk  Management  -  Under 
this  heading  would  be  the  intended  approach 
(specific  to  the  program)  for  executing  the 
processes  of: 

•  Risk  Assessment 

•  Risk  Analysis 

•  Risk  Handling. 

Also  appropriate  would  be  the  definitions, 
measurement  techniques,  and  risk  rating 
methods  for: 

•  Technical  Risk 

•  Programmatic  Risk 

•  Supportability  Risk 

•  Schedule  Risk 

•  Cost  Risk. 

A  description  of  the  structure  to  be  used 
to  identify  and  assess  project  risks  and  an  over¬ 
view  of  the  methods  and  techniques  for  risk 
analysis  would  be  valuable. 

Application  Issues  and  Problems  -  This 
section  would  include  the  procedures  and  proc¬ 
esses  for: 

•  Identifying  risks 

•  Quantifying  risk 

•  Use  of  tools  to  analyze  risk 

•  Applying  specific  actions  to 
manage  risk. 

Other  Relevant  Plans  -  Every  major  pro¬ 
gram  is  governed  by  a  set  of  plans  that  include: 

•  Program  Management  Plan 
(PMP) 

•  Systems  Engineering  Man¬ 
agement  Plan  (SEMP) 
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•  Acquisition  Plan  (AP) 

•  Test  and  Evaluation  Master 
Plan  (TEMP) 

•  Manufacturing  Plan  (MP) 

•  Integrated  Logistics  Sup¬ 
port  Plan  (ILSP). 

These  plans  provide  insights  into  items  of  risk. 
Typically  they  are  not  written  from  a  risk  view¬ 
point,  but  when  one  reads  them  with  an  eye  to 
raising  risk  questions,  they  provide  valuable  in¬ 
formation.  These  plans  should  be  reviewed  be¬ 
fore,  during,  and  after  preparation  of  the  Risk 
Management  Plan.  These  plans  may  suggest 
items  of  risk.  The  Risk  Management  Plan  may 
suggest  items  that  need  to  be  addressed  in  the 
other  plans.  While  the  Risk  Management  Plan 
deals  with  analyzing  and  managing  risk,  risk 
should  be  identified  and  highlighted  in  any 
or  all  plans  where  it  is  appropriate. 


43  RISK  ASSESSMENT 

43.1  Identification 

Risk  Identification  is  the  first  step  in  the 
risk  assessment  process.  Risks  cannot  be  as¬ 
sessed  or  managed  until  they  are  identified  and 
described  in  an  understandable  way.  Risk 
identification  is  an  organized  thorough  ap¬ 
proach  to  seek  out  the  real  risks  associated  with 
the  program.  It  is  not  a  process  of  trying  to  in¬ 
vent  highly  improbable  scenarios  of  unlikely 


events  in  an  effort  to  cover  every  conceivable 
possibility  of  outrageous  fortune. 

Approaches  -  Expert  interviews,  anal¬ 
ogy  comparisons,  and  the  evaluation  of  the 
program  plans  are  techniques  that  are  espe¬ 
cially  strong  in  the  risk  identification  segment. 
The  objective  of  the  risk  identification  segment 
is  to  obtain  straight  forward  English  language 
narrative  statements  describing  the  program 
risks.  Mathematical  techniques  are  not  appro¬ 
priate  here.  Chapter  5  describes  in  great  detail 
the  techniques  for  executing  the  risk  manage¬ 
ment  process  -  including  risk  identification. 

Baselining  Risk  -  Risk  exists  only  in  rela¬ 
tion  to  the  two  states  of  uncertainty  -  total  fail¬ 
ure  (usually  0%  probability)  and  total  success 
(usually  100%  probability).  The  risk  assess¬ 
ment  process  attempts  to  treat  risk  in  a  proba¬ 
bilistic  manner  and  the  process  is  significantly 
simplified  if  we  are  able  to  define  total  failure 
and  total  success.  Defining  one  or  both  of  the 
“baseline  programs”  is  worth  some  effort  in  or¬ 
der  to  obtain  a  benchmark  on  the  continuum 
(Figure  4.3-1).  It  is  certainly  desirable  but  dif¬ 
ficult  to  describe  the  technical  content  of  a  0 
percent  and  100  percent  probability  program. 
Usually,  however,  the  technical  content  is  given 
and  the  baseline  is  expressed  as  the  0%  and 
100%  probable  schedule  and  cost  values  to 
achieve  the  technical  content.  After  defining  a 
baseline  position,  it  becomes  easier  to  quantity 
risk  in  terms  of  each  impact  area  on  a  meaning¬ 
ful  scale. 
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Checklist  Concept  -  The  purpose  of  any 
program  is  to  achieve  a  specifiable  set  of  goals. 
The  basic  risk  identification  question  becomes, 
“What  are  the  events  or  facts  that  may  reason¬ 
ably  occur  v;hich  will  prevent  the  achievement 
of  program  goals?”  Occurrences  whose  out¬ 
comes  are  irrelevant  to  program  goals  have  no 
risk.  The  search  should  be  directed  toward  the 
“show  stoppers”  that  will  have  a  major  impact 
on  the  program.  The  key  to  risk  identification  is 
the  systematic  combing  through  the  total  pro¬ 
gram.  Figure  4.3-2  offers  a  matrix  that  can 
serve  as  a  tool  to  organize  this  process. 

The  Top  Level  Risk  Matrix  is  applied  at 
the  total  program  level  as  a  starting  point.  The 
concept  can  be  refined  and  carried  to  greater 
detail  as  needed. 

Defining  Program  Goals  -  One  would 
expect  this  step  to  be  an  easy  task.  More  than 
likely,  it  will  be  a  thought  provoking  and  con¬ 
troversial  process.  Requirements  specified  in 


thePMD  should  all  be  included  as  goals.  If  di¬ 
rection  is  missing  or  not  explicit  enough  to  be 
included  as  a  goal,  this  process  identifies  that 
fact  (which  in  itself  is  an  important  risk  reduc¬ 
tion  action).  All  goal  blocks  on  the  matrix 
should  be  covered.  A  goal  block  that  cannot 
be  filled  out  to  the  satisfaction  of  the  program 
manager  is  an  alert  for  direction  and/or  defini¬ 
tion.  The  program  manager  should  precipitate 
some  action  to  fill  the  void. 

Defining  Program  Strategies  -  Program 
strategies  represent  the  plan(s)  for  achieving 
the  goals.  In  the  ideal  case,  the  strategy  blocks 
in  the  matrix  should  contain  references  to 
chapters  or  paragraphs  in  one  or  more  of  the 
program  plans.  If  this  is  not  the  case,  the  plans 
are  inadequate.  This  causes  the  greatest  risk  of 
all  -  that  of  not  having  a  plan  to  reach  a  goal. 
The  Top  Level  Risk  Matrix  can  serve  as  a  forc¬ 
ing  function  to  insure  the  plans  address  all 
goals. 
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Identifying  Risks  -  A  simple  first  step  in 
risk  identification  is  to  evaluate  the  appropri¬ 
ateness  of  the  strategies  against  the  goals. 
Counterproductive  strategies  cause  risk.  The 
very  imperfect  world  of  systems  acquisition  fre¬ 
quently  forces  the  program  manager  to  do 
things  that  are  counterproductive  or  subop¬ 
timum.  Highlighting  these  anomalies  is  a  pow¬ 
erful  contribution  to  risk  identification. 

43.2  Preliminary  Quantification 

After  the  risk  identification  process  has 
produced  a  well  documented  description  of  the 
program  risks  and  before  risk  analysis  begins 
in  earnest,  some  organization  and  stratification 
of  the  identified  risks  is  beneficial.  Preliminary 
quantification  is  intended  to  provide  some 
prioritization  of  the  risks  for  further  evalu¬ 
ation.  Heavy  mathematical  treatment  is  not  de¬ 
sired  here. 

Rating  Schemes  and  Definitions  -  The 
degree  of  risk  existing  in  a  given  situation  is  a 
reflection  of  the  personality  of  the  risk  taker. 
Twenty  people  can  look  at  the  same  situation 
and  assign  twenty  different  risk  values  to  it.  A 
risk  rating  scheme  built  against  an  agreed  set  of 
definitions  provides  a  framework  for  eliminat¬ 
ing  some  of  the  ambiguity. 

The  rating  system  can  (and  probably 
should)  be  very  simple  -  such  as  High,  Medium, 
Low.  Using  the  notion  that  the  degree  of  risk  is 
a  judgment  reflecting  the  probability  of  occur¬ 
rence  and  the  severity  of  impact.  Figure  4.3-3 
offers  a  conceptual  diagram  for  a  risk  rating 


mechanism.  The  definition  issue  becomes  one 
of  identifying  impacts  and  deciding  on  a  scale 
(s)  and  then  shaping  the  boundaries  between 
the  three  regimes. 

With  a  defined  risk  rating  scheme  in 
place  (at  least  tentatively),  the  task  of  evaluat¬ 
ing  and  quantifying  each  of  the  identified  risks 
may  be  accomplished  against  this  structure. 

Interviewing  Experts  -  The  technique  of 
interviewing  experts  is  discussed  in  detail  in 
Section  5.2.  The  objective  is  to  gather  informa¬ 
tion  from  the  technical  experts  that  will  allow 
the  analyst  to  rate  the  risk. 

Using  Analogies  -  Analogy  comparison 
is  discussed  in  detail  in  Section  5.3.  It  is  an  at¬ 
tempt  to  learn  from  other  programs  or  situ¬ 
ations.  Analogy  comparison  is  a  technique 
used  for  many  things,  e.g.,  cost  estimating.  The 
caution  in  this  case  is  to  differentiate  between 
“analogous  programs”  and  “programs  with 
analogous  risks.” 


4.4  RISK  ANALYSIS 

4.4.1  Definition  and  Description 

The  transition  from  risk  assessment  ac¬ 
tivities  to  risk  analysis  activities  is  gradual. 
There  is  some  amount  of  analysis  that  occurs 
during  the  assessment  process.  For  example,  if. 
in  the  process  of  interviewing  an  expert,  a  risk 
area  is  identified,  it  is  logical  to  pursue  infor¬ 
mation  on  the  magnitude  of  the  risk,  the  con¬ 
sequences  if  the  risk  becomes  reality,  and  the 
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Figure  4 .3-3  Risk  Rating 


*  Could  be  cost,  schedule,  performance,  or  some  other  measurable 
factor.  May  also  be  combinations  or  multiple  scales  for  each 
parameter. 


possible  ways  of  dealing  with  it.  The  latter  two 
actions  are  generally  considered  a  part  of  the 
analysis  process  but  occur  during  the  risk  iden¬ 
tification  activities  of  a  formal  risk  manage¬ 
ment  effort.  This  is  illustrated  in  Figure  4.4-1. 

As  time  progresses  in  a  grass  roots  risk 
management  effort,  the  risk  analysis  function 
grows  independent  from  the  assessment  func¬ 
tion.  The  process  generally  becomes  more  of  a 
top  level  analysis  with  the  impacts  being  evalu¬ 
ated  against  total  project/program  completion 


or  subsystem  completion.  Risk  Analysis  in¬ 
volves  an  examination  of  the  change  in  conse¬ 
quences  caused  by  changes  in  the  risk  input 
variables.  Sensitivity  and  “what-if”  analysis  are 
examples  of  the  activities  that  should  take 
place  during  risk  analysis. 

4.4.2  Products  of  Risk  Analysis 

One  of  the  most  useful  products  of  the 
analysis  process  is  the  watchlist.  The  watchlist 
serves  as  the  worksheet  that  managers  use  for 
recording  risk  management  progress  (Ref  4.2). 
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Figure  4.4-1  Process  Time  Phasing 


An  example  of  a  watchlist  is  shown  in  Figure 
4.4-2.  Watchlists  provide  a  convenient  and 
necessary  form  to  track  and  document  activi¬ 
ties  and  actions  resulting  from  the  risk  analy¬ 
sis  process.  Cumulative  probability  distribu¬ 
tion,  another  useful  product  of  risk  analysis,  is 
illustrated  in  Figure  4.4-3.  The  cumulative 
probability  distribution  curve  is  a  common, 
conventional  method  used  to  portray  cost, 
schedule,  and  performance  risk.  Program  man¬ 
agement  offices  can  use  cumulative  probability 
distributions  by  determining  an  appropriate 
risk  level  (threshold)  for  the  item  and  reading 
from  the  curve  the  corresponding  target  cost, 
schedule,  or  performance.  This  is  a  typical  out¬ 
put  of  many  automated  risk  tools.  Appendix  E 
has  a  more  detailed  explanation  of  probability 
curves.  The  results  of  risk  analysis  are  ex¬ 
tremely  valuable  in  presentations  to  decision¬ 
makers.  The  process  of  performing  risk  analy¬ 


sis  generally  provides  an  in-depth 
understanding  of  the  sources  and  degree  of 
risk  and  can  be  quickly  portrayed  in  a  few 
charts.  This  provides  for  much  more  effective 
presentation/communication  to  decision¬ 
makers  of  the  program/project  status.  Section 
6.5  has  suggestions  for  communicating  risk  in¬ 
formation. 


4.5  RISK  HANDLING 

Risk  Handling  is  the  last  critical  ele¬ 
ment  in  the  risk  management  process.  It  is  the 
action  or  inaction  taken  to  address  the  risk  is¬ 
sues  identified  and  evaluated  in  the  risk  assess¬ 
ment  and  risk  analysis  efforts.  Generally,  these 
actions  fall  into  one  of  the  following  catego¬ 
ries. 

•  Avoidance 
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EVENT/ITEM  AREA  OF  IMPACT 


HANDLING  ACTION 


Loss  Of  Competition 

Production  Cost 

•  Breakout 

•  Qualify  2nd  Source 

•  Get  Tech  Data  as  a  Deliverable 

Incomplete  Logistic 

Support  Analysis 

Support  Cost 

•  Contractor  Support 

for  2-3  years 

•  Warranty  on  High  Risk  Items 

•  Emphasis  in  Contractor  Reviews 

•  Logistics  Reviews 

Immature  Tech  Data  Package  Production  Cost  with  High  1st  Unit 
with  many  Engineering  Cost  and  many  ECPS 

Changes  for  Design  Fixes 


Require  Production  Engineers 
on  Contractor  Design  Team 
Fixed  Price  Contract 
Competition 

Producibility  Engineering  Planning 
Production  Readiness  Reviews 


Long  Lead  Items  Delayed  Production  Schedule 


•  Get  Early  Identification  of  Long 

Lead  Item? 

•  Contractor  Emphasis  on 

Early  Delivery 

•  Transfer  or  Leveling  from  Less 

Urgent  Programs 

•  Buy  a  Position  in  Line 

for  Waiting 


Figure  4.4-2  Watchlist  Examples 


•  Control 

•  Assumption 

•  Transfer 

•  Knowledge  and  research. 

4.5.1  Risk  Avoidance 

The  statement  “I  do  not  accept  this  op¬ 
tion  because  of  the  potentially  unfavorable  re¬ 
sults”  reflects  what  is  meant  by  risk  avoidance. 
There  are  many  situations  where  a  lower  risk 
choice  is  available  from  several  alternatives. 
Selecting  the  lower  risk  choice  represents  a  risk 
avoidance  decision.  This  is  typical  of  the  evalu¬ 


ation  criteria  used  in  source  selection.  Cer¬ 
tainly,  not  all  risk  should  be  avoided  in  all  in¬ 
stances  though.  There  are  occasions  where  a 
higher  risk  choice  can  be  deemed  more  appro¬ 
priate  because  of  design  flexibility,  Pre- 
Planned  Product  Improvements  (P3I),  etc. 

4.5.2  Risk  Control 

This  is  the  most  common  of  all  risk  han¬ 
dling  techniques.  It  is  typified  by  the  statement 
“I  am  aware  of  the  risk,  and  I  will  do  my  best  to 
mitigate  it’s  occurrence  and  effect.”  Risk  con¬ 
trol  is  the  process  of  continually  monitoring 
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Figure  4.4-3  Cumulative  Probability  Distribution 


and  correcting  the  condition  of  a  program.  This 
often  involves  the  use  of  reviews,  risk  reduction 
milestones,  development  of  fallback  positions 
and  similar  management  actions.  Controlling 
risk  involves  the  development  of  a  risk  reduc¬ 
tion  plan  and  then  tracking  to  the  plan.  This  in¬ 
cludes  not  only  the  traditional  cost  and  sched¬ 
ule  plans,  but  also  technical  performance 
plans. 

4.5,3  Risk  Assumption 

Risk  Assumption  is  a  conscious  decision 
to  accept  the  consequences  should  the  event 


occur.  Some  amount  of  risk  assumption  is  al¬ 
ways  present  in  acquisition  programs.  The 
program  management  office  must  determine 
the  appropriate  level  of  risk  that  can  safely  be 
assumed  in  each  situation  as  it  is  presented.  An 
example  of  risk  assumption  is  permitting  pro¬ 
grams  to  have  significant  amounts  of  concur¬ 
rency. 

4.5.4  Risk  Transfer 

There  are  options  available  to  program 
offices  to  reduce  risk  exposure  by  sharing  risk. 
There  are  many  ways  to  share  risk  with  contrac¬ 
tors.  Type  of  contract,  performance  incentives. 
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warranties,  etc.  are  all  forms  of  sharing  risk 
with  contractors.  Note  that  many  of  these  only 
share  cost  risk.  Risk  transfer  is  often  beneficial 
to  both  the  contractor  and  the  government. 

4.5.5  Knowledge  and  Research 

While  this  is  not  a  “true”  risk  handling 
technique,  it  does  supply  the  other  methods 
with  valuable  information.  This  is  a  continuing 
process  that  enables  the  participants  to  per  ¬ 
form  risk  handling  (with  the  other  methods) 
with  greater  confidence.  It  consists  of  gathering 
additional  information  to  further  assess  risk 
and  develop  contingency  plans. 

Risk  handling  methods  are  only  con¬ 
strained  by  the  ingenuity  and  skills  contained 
within  the  program  office.  While  a  conscious 
decision  to  ignore  (assume)  a  risk  is  a  viable  op¬ 
tion,  an  unconscious  decision  to  do  the  same  is 
not.  A  documented  action  with  supporting  ra¬ 
tionale  is  recommended  ir.  all  risk  handling  op¬ 
tions.  (Ref.  4-3).  Note  that  the  risk  handling 
techniques  are  not  independent  of  each  other. 
For  example,  assuming  the  risk  involved  in  a 
concurrent  program  does  not  preclude  the  pro¬ 
gram  manager  from  instituting  measures  to 
control  inherent  risk. 


4.6  CHAPTER  4  KEY  POINTS 

•  Risk  management  is  the 
umbrella  function  for  the 
key  steps 


•  Risk  planning  sets  out  the 
requirements 

•  Risk  assessment  is  the  proc¬ 
ess  of  identifying  and  quan¬ 
tifying  program  risks  -  a 
well  aennea  rating  scheme 
is  critical 

•  Risk  analysis  is  the  process 
of  evaluating  program  im¬ 
pacts  as  a  results  of  risk  as¬ 
sessment 

•  Risk  handling  is  the  process 
of  executing  management 
actions  to  mitigate  or  elimi¬ 
nate  the  unwanted  results 
of  risk 

•  Risk  management  is  a  con¬ 
tinual  process  through  ail 
program  phases. 
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Chapter  5 

EXECUTING  THE  RISK  MANAGEMENT  PROCESS 


Having  gained  an  understanding  of  the 
concepts  of  risk  and  the  structure  useful  for 
executing  risk  management,  it  is  logical  to  now 
present  some  specific  techniques  that  apply  to 
the  process. 


5.1  INTRODUCTION 

All  processes  require  two  broad  cate¬ 
gories  of  action  (Figure  5.1-1): 

•  Planning 

•  Execution. 

This  chapter  covers  the  risk  manage¬ 
ment  techniques  that  have  proven  useful  to 
both  contractors  and  government  program  of¬ 
fices  in  the  execution  of  the  risk  management 
process.  The  planning  issues  were  covered  in 
Section  4.2  and  will  be  reiterated  in  Chapter  6. 
There  are  basically  seven  steps  in  the  execution 
portion  of  risk  management  as  outlined  below: 


1)  Evaluate  the  achievability 
of  the  proposed  project 
against  tneplan 

2)  Identify  the  risk  areas 

Develop  a  structure 
to  systematically  comb 
through  the  program  and 
issues  (i.e.,  WBS,  check¬ 
list) 

Interview  subject 
area  experts 

Review  analogous 
system  data 

Evaluate  the  program 
plans,  do  they  coincide? 

Examine  lessons 
learned  documents  (i.e., 
transition  templates,  stud¬ 
ies,  etc.) 

3)  Quantify  the  risk  areas 

Develop  a  consistent 
scheme  for  rating  risk. 
Make  it  quantitative  with 
qualitative  backup 

Assess  the  likelihood 
of  the  risk  occurring 
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EXECUTION  PHASE 


\  \  A 


z 

Ui 

x  2 

i  2!2 

</) 

< 


W  </> 

A  p  t" 

WD JznS 
00<i£2 
ujwUowO 
ui  tii  O  ui  uuu 
Zccu-koco: 


\  s§ 

J  s? 


§y 


iuw  tja 


XZUJUJUI 
UJ<  CC  JH 


/ 

/ 


is  »s 

|8  p 

?  ? 


iil 

isStfie 


Figure  5.1-1  The  Risk  Management  Process 


‘  -  Assess  the  impact  se¬ 
verity  in  terms  of  cost/ 
schedule/performance 

4)  Document  the  risk  areas 

Develop  and  main¬ 
tain  a  management 
watchlist 

Develop  an  effective 
communication  scheme  so 
input  from  all  functional 
areas  is  received 

5)  Utilize  an  analysis  tool  de¬ 
signed  to  meet  your  spe¬ 
cific  objectives.  Examine 
the  results: 

In  terms  of  perform¬ 
ance/time/cost 

By  system/subsystem 

Of  funding  profiles 

Based  on  criticality 

For  consistency  with 
analogous  systems 

Of  “what-if  ’  analysis 

6)  Determine  the  appropriate 
handling  option: 

Avoid  the  risk 

Share  the  risk  with 
another  party 

Assume  the  risk 

Control  the  risk 

7)  Implement  the  appropriate 
option. 


The  specific  techniques  for  accomplish¬ 
ing  these  steps  are  contained  in  the  following 
pages  of  this  chapter.  Many  of  the  techniques 
can  be  used  as  tools  for  multiple  parts  of  the 
process.  For  example,  an  in-depth  evaluation 


of  a  critical  path  network  is  very  useful  for  steps 
1»  2  and  5  above.  It  can  be  used  to  evaluate  and 
identify  risks  in  an  approach  and  serve  as  an 
excellent  analysis  tool.  Figure  5.1-2  illustrates 
which  techniques  have  application  in  more 
than  one  step  of  the  process.  The  predominant 
application  is  represented  by  a  solid  circle 
while  secondary  applications  are  represented 
by  a  hollow  circle. 
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KNOWLEDGE  A  RESEARCH  4.5.5 

Figure  5,1-2  Technique  Application 
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5.2  EXPERT  INTERVIEWS 

5.2.1  General 

One  of  the  most  critical  elements  or 
tasks  in  risk  assessment  is  that  of  obtaining  ac¬ 
curate  judgments  from  technical  experts.  Un¬ 
fortunately,  this  is  an  area  where  it  is  easy  to 
make  errors  and  therefore  obtain  information 
that  is  inaccurate.  The  interviewing  of  technical 
experts  to  gain  information  regarding  risk  is 
critical  for  two  reasons.  Hirst,  the  information 
identifies  those  areas  which  are  perceived  as 
being  risky  (risk  identification).  Second,  it  pro¬ 
vides  the  basis  for  taking  the  qualitative  infor¬ 
mation  and  transforming  it  into  quantitative 
risk  estimates  (risk  quantification).  Reliance 
on  the  advice  of  technical  experts  is  manda¬ 
tory  since  all  information  necessary  for  an  ac¬ 
curate  risk  assessment  usually  cannot  be 
derived  from  previous  program  data.  However, 
obtaining  the  information  from  experts  can  be 
frustrating  and  often  lead  to  less  than  optimum 
results. 

Nearly  all  risk  analysis  techniques  re¬ 
quire  some  form  of  expert  judgment  input. 
This  makes  the  acquisition  of  such  judgments 
extremely  important  to  the  overall  accuracy  of 
the  risk  management  effort.  As  previously 
mentioned,  this  is  a  very  difficult  task  to  per¬ 
form,  and  it  is  extremely  hard  to  distinguish 
between  “good"  and  “bad”  judgments.  This 
makes  the  approach  and  documentation  even 
more  important  than  usual.  The  program  man¬ 
ager  or  risk  analyst  performing  the  effort  is 
likely  to  get  several  divergent  opinions  from 


many  “experts”  and  he/she  must  be  able  to  de¬ 
fend  the  position  taken. 

5.2.2  Description  of  Technique 

The  expert  interview  technique  is  rela¬ 
tively  straightforward.  Basically,  it  consists  of 
identifying  the  appropriate  expert(s)  and  me¬ 
thodically  questioning  them  about  the  risks  in 
their  area  of  expertise  as  related  to  the  pro¬ 
gram.  There  are  many  methods  of  accomplish¬ 
ing  this  as  outlined  in  Appendix  F.  The 
technique  can  also  be  used  with  groups  of  ex¬ 
perts.  The  process  is  normally  aimed  at  obtain¬ 
ing  information  on  all  five  facets  of  risk. 

5.2 3  When  Applicable 

The  technique  is  useful  for  virtually  any 
program  and  is  recommended  for  all  programs. 
Expert  interviews  focus  on  extracting  informa¬ 
tion  about  what  the  program  risks  are  and  their 
relative  magnitude.  It  is  most  useful  in  the  risk 
assessment  portion  of  a  risk  management  ef¬ 
fort,  but  it  also  has  application  to  the  other 
processes  as  well.  When  questioning  experts 
about  the  risks  on  a  program,  it  is  logical  to 
pursue  potential  handling  actions  and  alterna¬ 
tives  as  well  as  information  pertaining  to  the 
potential  impact. 

5.2.4  Inputs  and  Outputs 

The  technique  has  two  prerequisites 
(required  as  input)  for  application.  First,  the  in¬ 
terviewer  must  be  prepared.  The  topic  must  be 
researched  and  an  interview  agenda  thought 
through.  Second,  the  interviewee  must  be  will- 
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ing  to  provide  the  information  sought  after 
and  be  willing  to  spend  the  necessary  time  re¬ 
quired  to  divulge  the  information  to  the  analyst 
or  manager.  The  results  (output)  of  such  inter¬ 
views  can  be  qualitative,  quantitative,  or  both. 
Expert  interviews  nearly  always  result  in  input 
that  can  be  used  in  the  formulation  of  a 
“watchlist”.  In  fact,  watchlists  frequently 
evolve  from  the  input  of  each  “expert”  func¬ 
tional  manager  on  a  program.  Another  fre¬ 
quently  useful  output  is  the  formulation  of  a 
range  of  uncertainty  or  a  probability  density 
function  for  use  in  any  of  several  risk  analysis 
tools.  These  can  be  in  terms  of  cost,  schedule, 
or  performance. 

5.2.5  Major  Steps  in  Applying 
the  Technique 

Since  expert  interviews  result  in  a  col¬ 
lection  of  subjective  judgments,  the  only  real 
“error”  can  be  in  the  methodology  for  collect¬ 
ing  the  data.  If  it  can  be  shown  that  the  tech¬ 
niques  for  collecting  the  data  are  not  adequate, 
then  the  entire  risk  assessment  can  become 
questionable.  Unfortunately,  there  is  no  sure 
fire  technique  for  assuring  that  the  data  col¬ 
lected  is  the  best  possible.  The  only  real  assur¬ 
ance  can  be  in  the  methodology  used  to  collect 
the  data.  There  are  several  methodologies 
available  for  collecting  data,  but  many  must  be 
ruled  out  because  of  the  time  restrictions  that 
usually  exist.  One  combination  (there  probably 
are  others  just  as  good)  which  seems  to  work 
well  consists  of  the  following  five  steps: 

•  Identify  the  right  individual 

•  Prepare  for  the  interview 


•  Target  the  interest  area 

•  Solicit  judgments  and  gen¬ 
eral  information 

•  Quantify  the  information. 

Each  of  these  steps  is  discussed  in  the 
following  paragraphs. 

Identify  the  Right  Individuals  -  It  is  ex¬ 
tremely  important  to  identify  the  correct  sub¬ 
ject  or  area  expert.  If  there  is  any  doubt  about 
the  level  of  expertise,  it  is  worthwhile  to  iden¬ 
tify  one  or  two  other  candidates.  It  is  relatively 
easy  to  make  a  mistake  in  this  area  by  identify¬ 
ing  an  expert  who  knows  only  a  portion  of  a 
given  area.  For  example,  if  you  are  interested 
in  knowing  the  risks  involved  in  the  test  pro¬ 
gram  for  a  particular  project  you  would  want  to 
talk  to  an  expert  in  the  test  field.  Someone 
who  knows  both  hardware  and  software  test 
procedures  would  be  appropriate.  The  time 
spent  up  front  identifying  the  individuals  to  be 
interviewed  will  be  well  spent.  Preliminary 
phone  screens  are  usually  worthwhile.  These 
usually  only  last  about  five  minutes  and  can 
give  the  analyst  a  feel  as  to  the  level  of  expertise 
an  individual  has  as  well  as  helping  to  focus  the 
questions  while  preparing  for  the  interview. 

Prepare  for  the  Interview-  A  lot  of  time 
can  be  saved  for  all  parties  if  there  has  been 
adequate  preparation  by  all  involved.  Some 
thought  should  be  given  as  to  what  areas  will 
be  covered  during  the  interview.  The  method¬ 
ology  for  quantifying  the  expert  judgment 
should  be  thoroughly  understood  and  re¬ 
hearsed  if  necessary.  It  is  much  easier  to  main- 
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tain  control  and  direction  during  the  interview 
if  there  is  an  agenda  or  list  of  topics  to  be  cov¬ 
ered.  It  is  also  helpful  to  understand  how  the 
individual  expert  functions  in  the  organization 
and  how  long  he  has  been  in  the  field.  It  is  nec¬ 
essary  to  keep  the  ultimate  goals  of  risk  identi¬ 
fication  and  quantification  in  mind  while 
preparing  for  the  interview.  This  means  that 
there  has  to  be  some  “open  time”  during  the 
interview  to  allow  the  expert  to  give  the  inter¬ 
viewer  his/her  personal  thoughts  on  areas 
which  may  be  outside  his/her  field. 

Target  the  Interest  Area  -  The  first  por¬ 
tion  of  the  actual  interview  should  be  to  focus 
on  the  previously  identified  risk  areas  to  ob¬ 
tain  verification.  This  should  be  kept  brief,  ex¬ 
cept  where  there  appears  to  be  a  conflict  which 
would  require  additional  information.  Next, 
the  interview  should  move  to  the  individual’s 
area  of  expertise.  This  will  either  confirm  that 
the  correct  individual  is  being  interviewed  or 
will  cause  the  focus  of  the  interview  to  change. 
By  targeting  the  interest  area  early,  more  time 
can  be  spent  within  the  individual’s  area  of  ex¬ 
pertise  if  necessary,  or  the  interview  can  be 
changed/ended  saving  valuable  time  if  there 
has  been  an  error  in  identifying  the  correct  in¬ 
dividual. 

Solicit  Judgments  and  General  Informa¬ 
tion  -  It  is  important  to  let  the  expert  have  some 
time  to  discuss  other  areas  of  the  program  if 
he/she  desires  after  completing  the  target  in¬ 
terest  areas.  If  nothing  else,  the  information 
gained  can  be  used  when  interviewing  in  an¬ 
other  area  to  stimulate  thoughts  and  generate 
another  opinion.  In  many  cases  an  “outside” 


observer  who  is  involved  in  the  program  can 
identify  potential  areas  of  conflict/risk  which 
may  not  be  apparent  to  the  person  working  in 
the  area  where  the  potential  conflict/risk  re¬ 
sides.  Much  of  the  initial  assessment  is  gained 
through  just  a  few  interviews.  This  information 
generally  becomes  more  refined/deleted/ex¬ 
panded  as  the  subject  experts  are  interviewed. 
Experience  has  shown  that  if  the  expert  is  co¬ 
operative,  the  information  given  (even  that 
which  is  outside  the  area  of  expertise),  is  gener¬ 
ally  correct.  Often  additional  clarification  is  re¬ 
quired  and  the  expert  is  unwilling  to  attempt  a 
quantification  but  the  identification  of  risk  is 
still  valid. 

Quantify  the  Information  -  This  is  the 
most  sensitive  aspect  of  any  risk  analysis.  Once 
the  risk  areas  have  been  identified,  an  estimate 
of  their  potential  impact  on  the  program  per¬ 
formance,  cost,  and  schedule  must  be  made. 
This  requires  that  the  expert  consider  the  prob¬ 
ability  of  the  given  risky  event  occurring,  and 
what  the  potential  impact  may  be  in  terms  of 
performance,  cost,  and  schedule. 

§.2.6  Use  of  Results 

Normally,  the  results  of  expert  inter¬ 
views  feed  other  techniques  or  are  used  in  the 
development  of  watchlists  as  described  in  Sec¬ 
tion  4.4.2. 

5.2.7  Resource  Requirements 

Interviewing  experts  requires  two  spe¬ 
cific  resources.  The  first  of  which  is  time.  While 
this  is  one  of  the  most  common  techniques  in 
use  for  risk  assessment,  it  is  also  one  which  is 
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frequently  misapplied  because  of  time  limita¬ 
tions.  Planned  interviews  are  sometimes  short¬ 
ened  or  skipped  altogether  in  order  to  meet 
other  obligations  or  deadlines  by  the  inter¬ 
viewer  and  interviewee.  A  methodical  exami¬ 
nation  of  an  entire  program  requires  the  time 
of  many  experts  -  both  from  the  government 
and  contractor.  The  second  resource  require¬ 
ment  is  an  experienced  interviewer.  Fre¬ 
quently,  experts  do  not  give  information  which 
is  readily  usable  for  a  watchlist  or  probability 
density  function.  Some  skill  is  required  to  en¬ 
courage  the  expert  to  divulge  information  in 
the  right  format.  If  an  experienced  interviewer 
is  not  available,  the  technique  can  still  yield 
some  valuable  information  if  enough  time  is  al¬ 
located. 

5.2.8  Reliability 

When  conducted  properly,  expert  inter¬ 
views  provide  very  reliable  qualitative  informa¬ 
tion.  The  transformation  of  that  qualitative 
information  into  quantitative  distributions  or 
other  measures  depends  on  the  skill  of  the  in¬ 
terviewer.  The  technique  is  not  without  prob¬ 
lems.  Some  typical  problems  that  experienced 
risk  analysts  have  had  are  listed  below. 

•  Wrong  expert  identified 

•  Poor  quality  information 
obtained 

•  Unwillingness  of  the  ex¬ 
pert  to  share  information 

•  Changing  opinions 

•  Conflicting  judgments 


53  ANALOGY  COMPARISON/ 

LESSONS  LEARNED  STUDIES 

53.1  General 

The  “analogy  comparison”  and  “lessons 
learned”  techniques  for  risk  identification  and 
assessment  are  based  on  the  idea  that  no  new 
program,  no  matter  how  advanced  or  unique, 
represents  a  totally  new  system.  Most  “new" 
programs  originated  or  evolved  from  already 
existing  programs  or  simply  represent  a  new 
combination  of  existing  components  or  subsys¬ 
tems.  Alogical  extension  of  this  premise  is  that 
key  insights  can  be  gained  concerning  the  vari¬ 
ous  aspects  of  a  current  program’s  risk,  by  ex¬ 
amining  the  successes,  failures,  problems,  and 
solutions  of  similar  existing  or  past  programs. 
The  experience  and  knowledge  gained,  or  “les¬ 
sons  learned”  can  be  applied  to  the  task  of 
identifying  potential  risk  in  a  program  and  de¬ 
veloping  a  strategy  to  handle  that  risk. 

53.2  Description  of  Technique 

The  analogy  comparison  and  lesson 
learned  techniques  involve  the  identification 
of  pastor  existing  programs  that  are  similar  to 
the  Program  Management  Office  (PMO)  effort 
and  the  review  and  use  of  data  from  these  pro¬ 
grams  in  the  PMO  risk  management  process. 
The  term  “similar”  refers  to  the  commonality 
of  the  variety  of  characteristics  which  defines  a 
program.  The  analogy  may  be  similar  in  tech¬ 
nology,  function,  acquisition  strategy,  manu¬ 
facturing  process,  etc.  The  key  is  to  understand 
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the  relationship  between  the  program  charac¬ 
teristics  and  the  particular  aspect  of  the  pro¬ 
gram  being  examined.  For  example,  in  many 
system  developments,  historic  cost  data  shows 
a  strong  positive  relationship  with  technical 
complexity.  Thus  when  looking  for  a  program 
in  which  to  analyze  cost  risk  for  comparison,  it 
makes  sense  to  examine  data  from  programs 
with  similar  function,  technology,  and  techni¬ 
cal  complexity.  The  use  of  data  or  lessons 
learned  from  past  programs  may  be  applicable 
at  the  system,  subsystem  or  component  level. 
For  example,  though  an  existing  system’s  func¬ 
tion  and  quantity  produced  differ,  its  processor 
may  be  similar  in  performance  characteristics 
to  a  current  program  and  thus  a  valid  basis  for 
analogy  comparison.  Several  different  pro¬ 
grams  may  be  used  for  comparison  to  the  cur¬ 
rent  project  at  various  levels  of  the  end  item. 

5.33  When  Applicable 

The  application  of  documented  lessons 
learned  or  the  comparison  of  old  or  existing 
programs  to  new  programs  is  useful  in  all 
phases  and  aspects  of  a  program.  In  any  situ¬ 
ation  in  which  historic  data  is  useful  in  predict¬ 
ing  or  anticipating  the  future,  the  analogy 
comparison  and  lessons  learned  technique  can 
provide  valuable  insights  into  the  risk  associ¬ 
ated  with  a  program.  These  techniques  are  es¬ 
pecially  valuable  when  a  new  system  is 
primarily  a  new  combination  of  existing  sub¬ 
systems,  equipment,  or  components  for  which 
recent  and  complete  historical  program  data 
is  available.  When  properly  done  and  docu¬ 
mented,  analogy  comparison  provides  a  good 
understanding  of  how  the  program  character¬ 


istics  affect  the  risk  identified  and  provide  a 
necessary  input  to  many  other  risk  techniques. 

53.4  Inputs  and  Outputs 

There  are  three  types  of  data  required 
for  use  of  the  technique: 

•  Description  and  program 
characteristics  of  the  new 
system  and  its  components 

•  Description  and  program 
characteristics  of  the  exist¬ 
ing  or  past  programs  and 
their  components 

•  Detailed  data  for  the  prior 
system  being  reviewed 
(cost,  schedule,  perform¬ 
ance,  etc). 

The  descriptive  data  and  the  program 
characteristics  information  is  needed  to  draw 
valid  analogies  between  the  current  and  past 
programs.  The  detailed  data  is  required  to 
evaluate  and  understand  program  risks  and 
their  potential  effect  on  the  current  project. 

Often  technical  specialists  are  needed 
to  help  make  appropriate  comparisons  and  to 
help  extrapolate  or  adjust  the  data  from  old 
programs  to  make  inferences  about  new  pro¬ 
grams.  Technical  or  program  judgments  may¬ 
be  needed  to  adjust  findings  and  data  for  dif¬ 
ferences  in  complexity,  performance,  physical 
characteristics  or  acquisition  approaches. 

The  output  from  the  examination  of 
analogous  programs  and  lessons  learned  typi¬ 
cally  becomes  the  input  to  other  risk  assess¬ 
ment  and  analysis  techniques.  The  review  of 
program  lessons  learned  reports  can  identify  a 
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number  of  problems  to  be  integrated  into  a 
program’s  watchlist.  The  length  and  volatility 
of  past  flight  test  programs  is  information  that 
would  aid  in  the  development  of  realistic  dura¬ 
tions  in  a  network  analysis  of  a  new  program’s 
test  schedule.  Data  from  the  review  of  lessons 
learned  and  past  analogous  programs  becomes 
the  source  of  information  for  the  conduct  of 
risk  assessment,  analysis,  and  handling  tech¬ 
niques. 

5 .3.5  Mtyor  Steps  in  Applying 

the  Technique 

The  major  steps  in  the  use  of  analogous 
system  data  and  lessons  learned  include  the 
identification  of  analogous  programs,  data  col¬ 
lection,  and  analysis  of  the  data  gathered.  Fig¬ 
ure  5.3-1  shows  a  further  breakdown  of  this 
process. 

The  first  step  is  to  determine  the  infor¬ 
mation  needs  in  this  phase  of  risk  management 
process.  This  could  vary  from  wanting  to  assess 
the  risk  involved  with  the  development  of  a  cus¬ 
tom  computer  chip  for  a  new  application  to  a 
broad  goal  of  identifying  all  of  the  major  risks 
associated  with  a  program. 

The  second  step  is  to  define  the  basic 
characteristics  of  the  new  system.  This  is  neces¬ 
sary  in  order  to  identity  past  programs  that 
are  similar  in  technology,  function,  design, 
etc.  With  the  new  system  generally  defined  the 
analyst  can  begin  to  identify  programs  with 
similar  attributes  for  comparison  and  analysis. 

The  next  steps  in  this  process,  being  in¬ 
terdependent,  are  generally  done  in  parallel. 


The  key  to  the  usefulness  of  analogy  compari¬ 
son  is  the  availability  of  data  on  past  programs. 
The  new  system  is  broken  down  into  logical 
components  for  comparison,  while  assessing 
the  availability  of  historical  data.  There  is  no 
use  in  analyzing  a  system  at  a  detailed  compo¬ 
nent  level  against  past  efforts  if  that  same  level 
of  detailed  information  is  not  available  in  past 
programs.  Based  on  the  availability  of  data, 
the  information  needs  of  the  process,  and  the 
logical  structure  of  the  program,  analogous 
systems  are  selected  and  data  gathered. 

The  data  gathered  for  comparison  in¬ 
cludes  the  detailed  information  being  analyzed 
as  well  as  the  general  characteristics  and  de¬ 
scriptions  of  the  past  programs.  The  general 
program  descriptive  data  is  essential  to  insure 
proper  analogies  are  being  drawn  and  a  clear 
understanding  of  the  relationship  between 
these  characteristics  and  the  detailed  data  be¬ 
ing  gathered  is  understood.  For  the  analogy  to 
be  valid,  there  must  be  some  relationship  be¬ 
tween  the  characteristic  being  used  to  make 
comparisons  and  the  specific  aspect  of  the  pro¬ 
gram  being  examined.  For  example,  if  there  is 
no  basis  for  relating  weight  to  schedule,  weight 
of  the  system  is  a  suspect  basis  for  drawing  an 
analogy  while  doing  a  schedule  assessment. 

Often  the  data  collection  process  and 
initial  assessment  leads  to  a  further  definition 
of  the  system  for  the  purposes  of  comparison. 
After  this  has  been  accomplished,  the  last  step 
in  the  process  is  the  analysis  and  normalization 
of  the  historic  data.  Comparisons  to  older  sys¬ 
tems  may  not  be  exact  or  the  data  may  need  to 
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Figure  5 .3-1  Analogy  Comparison 


be  adjusted  to  be  used  as  a  basis  for  estimating 
the  future.  For  example,  in  analogy  based  cost 
estimating,  cost  data  must  be  adjusted  for  in¬ 
flation,  overhead  rates,  G& A,  etc.  for  accurate 
comparison.  Technical  assistance  is  frequently 
needed  to  adjust  the  data  for  differences  in  past 
versus  the  current  program.  The  desired  out¬ 
put  is  some  insight  into  the  cost,  schedule,  and 


technical  risks  of  a  program  based  on  observa¬ 
tions  of  similar  past  programs. 

53.6  Use  of  Results 

As  stated  earlier,  the  output  from  anal¬ 
ogy  comparison  or  the  review  of  lessons 
learned  typically  feed  other  risk  techniques. 
The  results  may  provide  a  checklist  of  factors  to 
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monitor  for  the  development  of  problems  or  a 
range  of  cost  factors  for  use  in  estimating  (for 
example,  software  lines  of  code).  The  results 
of  analogy  comparison  and  lessons  learned  is 
risk  information.  Whether  the  information  is 
used  in  a  detailed  estimate,  technology  trade¬ 
off  study  or  at  a  system  level  for  a  quick  test  of 
reasonableness,  the  results  are  intended  to 
provide  the  analyst  with  information  on  which 
to  conduct  analyses  and  ultimately  base  deci¬ 
sions. 

5.3.7  Resource  Requirements 

The  use  of  analogous  data  and  lessons 
learned  studies  to  gather  risk  data  is  a  rela¬ 
tively  easy  task.  The  selection  of  proper  com¬ 
parisons  and  the  analysis  of  the  data  gathered 
may  require  some  technical  assistance  and 
judgment,  but  probably  not  beyond  the  capa¬ 
bilities  of  the  Program  Management  Office. 
The  time  and  effort  to  accomplish  an  analogy 
comparison  however,  can  vary  widely.  The  re¬ 
sources  needed  are  dependent  on  the  depth  of 
the  data  gathering,  the  number  of  different 
programs,  and  the  availability  of  historic  data. 
Much  effort  can  be  expended  gathering  a  little 
information.  That  is  why  an  initial  assessment 
of  data  availability  is  important  in  the  selection 
of  analogous  programs  for  comparison. 

5.3.8  Reliability 

There  are  two  limitations  to  the  use  of 
analogy  comparisons  and  lessons  learned.  The 
first,  the  availability  of  data,  has  already  been 
discussed.  The  absence  of  program  character¬ 
istics  or  detailed  data  about  the  new  or  old  sys¬ 


tem  limits  the  usefulness  of  the  data  collected. 
The  second  limitation  deals  with  the  accuracy 
of  the  analogy  drawn.  An  older  system  maybe 
somewhat  similar,  but  rapid  changes  in  tech¬ 
nology,  manufacturing,  etc.,  may  make  com¬ 
parisons  to  past  programs  inappropriate. 

5.4  PLAN  EVALUATION 

5.4.1  General 

This  technique  is  directed  at  highlight¬ 
ing  and  isolating  risks  caused  by  disparities  in 
planning.  It  evaluates  program  plans  for  con¬ 
tradictions  and  voids.  The  term  “plan”  as  used 
in  this  case  means  the  traditional  formal  plans 
to  govern  the  acquisition  of  a  major  system. 
These  include: 

•  Program  Management  Plan 
(PMP) 

•  Systems  Engineering 
Management  Plan 
(SEMP) 

•  Acquisition  Plan  (AP) 

•  Test  and  Evaluation 
Master  Plan  (TEMP) 

•  Manufacturing  Plan  (MP) 

•  Integrated  Logistics  Sup¬ 
port  Plan  (ILSP) 

Other  documents,  not  normally  thought 
of  as  plans,  but  key  to  the  success  of  a  program 
are: 

•  Work  Breakdown  Structure 
(WBS)  Index  and  Diction¬ 
ary 

•  Specifications  and  the 
Specification  Tree 
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•  Statements  of  Work 

•  Other  “Baseline”  Docu¬ 
ments 

While  the  first  group  of  plans  document 
the  steps  in  the  execution  of  the  program,  the 
latter  represent  the  absolutely  critical  commu¬ 
nication  with  the  contractor(s)  about  what  is  to 
be  done.  Flaws,  inconsistencies,  contradic¬ 
tions,  and  voids  in  these  documents  guarantee 
program  problems  and  introduce  significant 
risk.  Figure  5.4-1  illustrates  the  linkage  be¬ 
tween  the  three  key  documents. 

5.4.2  Description  of  Technique 

This  technique  simply  suggests  a  thor¬ 
ough  recurring  review  of  all  plans: 

•  Internally  for  correctness, 
completeness,  and  currency 

•  Cross  check  for  consis¬ 
tency. 

Using  the  Work  Breakdown  Structure  for 
Risk  Identification  -  The  proper  development 


of  a  WBS  represents  in  itself  a  major  step  in 
risk  avoidance.  It  constitutes  much  of  the  pro¬ 
gram  definition.  Its  quality,  indeed  its  very  ex¬ 
istence,  provides  the  framework  for  planning 
that  sets  the  standard  for  the  future  of  the  pro¬ 
gram. 

The  end  result  of  the  WBS  develop¬ 
ment  process  is  the  Project  WBS.  A  careful 
questioning  of  the  Project  WBS  is  appropriate. 

•  Are  all  elements  of  the 
WBS  necessary  and  suffi¬ 
cient? 

•  Is  there  a  WBS  dictionary 
and  does  it  adequately  ex¬ 
plain  the  content  of  each 
element? 

•  Does  the  WBS  represent 
what  is  to  be  done  rather 
than  who  is  to  do  it? 

•  Are  all  elements  of  the 
project  WBS  present? 

Summary  WBS 

Project  Summary  WBS 

Contract  WBS 

Contractor  Extension 
of  the  Contract  WBS. 


Figure  5.4-1  Plan  Evaluation  Technique 
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•  Is  the  procurement  strategy 
reflected  in  the  project 
WBS? 

•  Is  there  any  “work”  to  be 
done  that  is  not  in  the 
WBS? 

The  WBS  offers  a  framework  for  or¬ 
ganizing  and  displaying  risk  factors.  The  tech¬ 
nique  of  downward  allocation  and  upward 
summarization  through  the  WBS  can  be  used 
to  highlight  discrepancies  in  most  of  the  pro¬ 
gram’s  performance  parameters  such  as 
weight,  electrical  power,  cooling  requirements, 
system  reliability,  and  cost. 

The  WBS  provides  a  sensible  structure 
for  treating  technical  risk.  A  systematic  review 
of  each  WBS  element  for  risk  identification 
and  preliminary  rating  as  discussed  in  Section 
4.3  will  yield  much  information  to  the  risk  ana¬ 
lyst. 

The  relationship  between  the  Work 
Breakdown  Structure  and  the  Specification 
Tree  is  so  important  that  mapping  the  relation¬ 
ship  is  a  valuable  exercise  for  the  risk  analyst. 
The  mapping  will  highlight  inconsistencies  be¬ 
tween  the  “work  to  be  done”  and  the  “per¬ 
formance  to  be  achieved”. 

Figure  5.4-2  illustrates  the  fact  that  the 
project  WBS  eventually  becomes  the  aggregate 
of  contract  WBSs  and  the  contractor’s  exten¬ 
sion  thereof  which  includes  subcontractors 
WBSs.  The  risk  analyst  should  review  the  WBS 


with  the  question  “who  is  doing  what?”  as  a  test 
of  reasonableness  of  the  procurement/con¬ 
tracting  strategy.  Finally,  the  WBS  represents  '  " 
the  framework  for  cost  and  schedule  perform-  * 
ance.  A  survey  of  both  the  cost  and  the  sched-  , 
ule  reporting  against  the  WBS  identifies 
possible  blind  spots  in  cost  and  schedule  infor¬ 
mation.  As  part  of  this  survey,  the  analyst  can 
gain  valuable  insights  by  comparing  the  num¬ 
bering  schemes  for  the  WBS,  the  scheduling 
system(s),  and  the  cost  reporting  system(s). 
Ease  of  translation  between  and  ease  of  sum¬ 
marization  within  each  of  these  numbering  sys¬ 
tems  is  an  indicator  of  how  well  traceability 
among  the  WBS,  schedules,  and  cost  data  can 
be  maintained.  Incompatibility  introduces 
management  risk  into  the  program. 

Using  Specifications  and  the  Specifica¬ 
tion  Tree  for  Risk  Identification  -  Some  of  the 
discussion  above  deals  with  the  very  impor¬ 
tant  relationship  between  the  WBS  and  the 
Spec  Tree  and  the  need  for  compatibility. 
When  that  compatibility  exists,  it  is  possible  to 
reiate  the  performance  to  be  achieved  to  the 
work  to  be  done.  Since  the  specifications  rep¬ 
resent  the  source  of  all  technical  performance 
requirements,  they  are  the  single  most  impor¬ 
tant  source  of  information  for  the  risk  analyst 
attempting  to  identify,  organize,  and  display 
items  of  technical  risk.  Each  performance  pa¬ 
rameter  of  a  given  WBS  element  represents  a 
possible  focus  for  an  expert  interview  on  tech¬ 
nical  risk. 
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Figure  5.4-2  WBS  Preparation/Development 


As  with  the  WBS,  a  survey  of  the  speci¬ 
fications  and  the  specification  tree  is  appropri¬ 
ate  for  risk  identification. 

•  Does  the  Spec  Tree  overlay 
the  WBS  so  that  perform¬ 
ance  requirements  are 
specified  for  “whole”  WBS 
elements? 

•  Are  all  performance  pa¬ 
rameters  identified  even 
though  they  may  not  be 
specified  (i.e.,  given  a  dis¬ 
crete  value)? 

•  Is  it  possible  to  sensibly 
discuss  the  risk  of  achiev¬ 
ing  the  specified  value  for 
the  performance  parame¬ 
ter? 

•  Is  there  a  technical  per¬ 
formance  measurement 
scheme  for  each  perform¬ 
ance  parameter? 

Using  Statements)  of  Work  for  Risk 
Identification  -  The  Statement  of  Work  is  the 
single  most  important  communication  between 
the  program  manager  (who  wants  results)  and 
the  contractor  (who  has  to  produce  the  results). 
If  the  WBS  and  the  specifications  are  complete 
and  well  done,  statements  of  work  are  fairly 
straight  forward.  The  risk  analyst  is  primarily 
searching  for  gaps  in  coverage,  (i.e.,  work  and 
performance  requirements  that  have  not  been 
assigned  to  someone  (contractor)). 

•  Do  the  SOWs  cover  whole 
ieces  of  the  WBS  that  can 
e  evaluated  against  whole 

specifications? 

•  Do  the  SOWs  represent 
work  that  can  be  con¬ 
tracted  in  a  straight  for¬ 
ward  manner  or  will  the 
contracts  be  politically,  le¬ 


gally,  or  contractually  dif¬ 
ficult  to  execute  and 
manage? 

•  Is  all  of  the  work  contrac¬ 
tually  covered? 

•  Are  the  SOW  require¬ 
ments  properly  related  to 
the  specifications? 

Developing  a  Technical  Risk  Dictionary  - 
The  concept  of  dictionaries  is  understood  and 
fairly  well  institutionalized  in  DoD  acquisition 
program  offices.  The  WBS  dictionary  is  well 
known  and  well  established.  More  recently, 
program  offices  are  using  the  idea  of  a  sched¬ 
ule  dictionary  to  provide  the  definition  of  the 
activities  in  the  program  schedule  and  the  as¬ 
sumption  that  leads  to  their  durations. 

This  Section  5.4.2  has  thus  far  dealt  with 
a  body  of  information  that  represents  the  docu¬ 
mented  description  of  the  sum  and  substance  of 
an  acquisition  program.  A  technical  risk  dic¬ 
tionary  as  conceptualized  in  Figure  5.4-3  of¬ 
fers  a  way  for  the  risk  analyst  to  gather  this 
information  in  a  single  place  in  order  to  facili¬ 
tate  the  risk  identification/definition  process. 

The  creation  of  a  technical  risk  diction¬ 
ary  would  have  been  a  formidable  editorial 
task  until  recently.  Current  word  processing 
and  database  management  software  should 
make  the  bulk  of  the  task  one  of  electronic  cut 
and  paste.  Indeed,  if  document  and  paragraph 
numbering  are  done  with  a  view  of  interchan¬ 
geability  of  data,  the  technical  risk  dictionary 
could  be  quickly  created  with  a  single  utility 
program.  This  of  course  applies  to  the  technical 
content,  performance,  and  task  sections  of  the 
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Figure  5.4-3  Technical  Risk  Dictionary 
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dictionary  which  serve  as  background  material 
for  the  risk  section.  The  risk  section  represents 
original  thought  contained  only  in  this  docu¬ 
ment. 

Defense  Systems  Management  College 
is  engaged  in  an  effort  to  develop  automated 
tools  for  the  program  manager.  Two  of  these, 
the  Automated  Program  Planning  Documen¬ 
tation  Model  (APPDM),  and  the  Procurement 
Document  Generator  (PDG)  are  intended  to 
aid  in  the  creation  and  maintenance  of  the 
large  volumes  of  textual  material  required  by  a 
typical  program  office.  One  of  the  elements  of 
APPDM  is  a  model  Risk  Management  Plan 
(discussed  in  Section  4.2).  An  extension  of  this 
capability  to  produce  a  technical  risk  dictionary 
is  easily  within  reach. 

Using  Other  Plans  for  Risk  Identification  - 
In  Section  4.3.1,  the  use  of  a  Top  Level  Risk 
Matrix  to  highlight  and  isolate  risks  was  dis¬ 
cussed.  It  relies  heavily  on  goal  definition  and 
strategy  development.  The  presumption  is  that 
the  strategies  expressed  in  the  program  plans 
are  directed  at  meeting  the  program  goals. 
Comparing  the  two  is  a  way  to  identify  risks. 
The  same  thought  process  can  be  applied  to 
produce  lower  level  risk  matrices  for  each  of 
the  respective  plans  (e.g.,  the  TEMP  in  FSD). 

Some  particularly  astute  program  man¬ 
agers  aie  formally  including  discussions  of  risk 
within  the  program  plans  (as  it  should  be)  - 
either  as  a  section  in  each  chapter  or  as  a  sepa¬ 
rate  chapter. 


Summary 

In  the  ideal  world,  where  a  program 
management  office  is  staffed  with  seasoned 
professionals  of  long  tenure,  the  Plan  Evalu¬ 
ation  technique  would  produce  very  little  re¬ 
sults  for  a  large  effort.  All  of  the  planning 
documents  would  have  been  created  in  the 
proper  sequence;  each  with  reference  to  all 
that  preceded  it.  Eminently  logical  contracts 
would  have  been  let  with  masterful  work  state¬ 
ments  and  perfect  specifications.  In  reality,  ten¬ 
ure  in  a  program  office  is  very  short,  planning 
documents  are  prepared  simultaneously  or  out 
of  order  by  a  cast  of  people  having  a  wide  range 
of  experience,  both  totally  and  within  the  par¬ 
ticular  program.  Corporate  memory  is  very 
short  and  in  the  early  stages  when  most  of  the 
planning  is  accomplished,  most  program  man¬ 
agement  offices  are  grossly  undermanned. 
Therefore,  the  Plan  Evaluation  technique  is 
very  useful  in  program  management. 

5  43  When  Applicable 

This  technique  is  specifically  directed  at 
risk  identification.  It  is  best  used  for  technical, 
programmatic,  and  supportability  risk  identifi¬ 
cation.  Its  utility  for  cost  and  schedule  risk  is 
considerably  less.  However,  this  technique 
could  indicate  any  missing  information  con¬ 
cerning  deliverables  which  would  impact  cost 
and  schedule  risks.  It  is  most  applicable  to  the 
full  scale  development  and  production  phases 
of  a  program.  As  a  risk  identification  tech¬ 
nique,  it  requires  the  existence  of  the  plans  to 
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be  evaluated.  As  a  risk  avoidance  tool,  it  can  be 
used  during  the  program  planning  process. 

5.4.4  Inputs  and  Outputs 

The  technique  operates  on  the  collec¬ 
tive  body  of  documentation  broadly  referred  to 
as  “program  plans”.  This  includes  primarily 
those  documents  listed  in  Section  5.4.1.  The 
output  of  the  technique  will  typically  be: 

•  A  top  level  risk  matrix 

•  Lower  level  risk  matrices 

•  A  technical  risk  dictionary 

•  Updated  version  of  the 
program  plans 

5.4.5  M^jor  Steps  in  Applying 
the  Technique 


•  Evaluate  WBS: 

Completeness 

Correctness 

•  Evaluate  Spec  Tree: 

Completeness 

Correctness 

Compatibility  with  WBS 

•  Evaluate  SOWs: 

Completeness 

Correctness 

Compatibility  with  WBS 
Inclusion  of  spec 
references 

•  Other  plans: 

Develop  lower  level 
risk  matrix  for  each. 

5.4.6  Use  of  Results 

The  results  of  this  technique  are  best 
used  to  improve  the  quality  and  reduce  the 
risks  contained  in  the  program  plans.  The 
technique  also  produces  descriptive  documen¬ 


tation  of  the  technical,  performance,  program¬ 
matic,  and  supportability  risks  associated  with 
the  program.  The  technical  risk  dictionary  de¬ 
scribes  the  technical  risks  and  isolates  their  lo¬ 
cation.  The  program  manager  should  use  this 
technique  to  produce  a  single,  more  or  less  “of¬ 
ficial”  list  of  program  risks  that  will  receive  ac¬ 
tive  management  attention  (i.e.,  a  Watchlist). 

5.4.7  Resource  Requirements 

This  technique  requires  a  great  deal  of 
thought.  It  requires  experienced,  knowledge¬ 
able  personnel  who  are  intimately  familiar  with 
the  content  of  the  total  program.  The  deputy 
program  manager  leading  a  small  team  of  sen¬ 
ior  individuals  probably  represents  the  best 
means  of  executing  this  technique. 

5.4.8  Reliability 

The  reliability  of  this  technique  is 
driven  by  the  completeness  and  far¬ 
sightedness  of  the  program  plans.  The  relation¬ 
ship  is  an  inverse  one  -  the  better  the  plans,  the 
fewer  the  planning  risks  uncovered. 

The  major  caution  for  the  user  of  this 
technique  is  to  not  try  to  force  detailed  pro¬ 
gram  definition  too  eorly.  Some  inconsisten¬ 
cies  exist  because  of  poor  planning,  others  due 
to  a  legitimate  lack  of  information. 

5.5  TRANSITION  TEMPLATES 

5.5.1  General 

This  technique  is  based  on  the  work  per¬ 
formed  by  the  Task  Force  on  “Transition  from 
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Development  to  Production”.  Their  efforts  re¬ 
sulted  in  publication  of  DoDD  4245. 7-M, 
“Transition  from  Development  to  Produc¬ 
tion...  Solving  the  Risk  Equation”,  September 
1985.  This  manual  is  recommended  reading 
for  all  program  managers.  It  includes  extensive 
work  on  the  identification  of  program  pitfalls 
based  on  solid  experience.  The  focus  of  the 
book  is  on  disciplined  engineering  and  its  im¬ 
pact  on  the  entire  management  process 
through  all  phases  of  a  program.  There  is  also 
a  companion  manual,  NAVSO  P-6671,  “Best 
Practices,  How  to  Avoid  Surprises  in  the 
World’s  Most  Complicated  Technical  Proc¬ 
ess",  November  1985.  This  second  document 
identifies  specific  practices  in  use  and  their  po¬ 
tentially  adverse  consequences.  The  book  then 
describes  the  “best  practices”  which  avoid  or 
alleviate  these  consequences. 

5.5.2  Description  of  Technique 

The  technique  consists  of  examining  a 
sei  ies  of  “templates”  that  cover  specific  areas 
that  may  present  technical  risk  to  a  program. 
Each  template  examines  an  area  of  risk  and 
then  describes  methods  for  avoiding  or  reduc¬ 
ing  that  risk.  Much  of  the  description  of  the 
risk  and  the  solution  is  based  on  lessons 
learned  from  other  programs.  The  areas  cov¬ 
ered  by  the  templates  is  illustrated  in  Figure 
5.5-1. 

5.5.3  When  Applicable 

This  technique  should  be  used  for  most 
programs  -  either  independently  or  in  con¬ 
junction  with  another  technique.  The  informa¬ 


tion  contained  within  the  templates  is 
extremely  valuable  to  all  program  managers 
because  it  is  based  on  actual  experiences.  The 
information  can  be  useful  for  any  size  program 
at  any  phase  of  development.  Since  the  tech¬ 
nique  views  the  acquisition  process  as  a  com¬ 
plete  process  (that  is  design,  test,  and 
production  are  integral  parts  of  a  whole  sys¬ 
tem),  the  solutions  presented  reflect  the  inter¬ 
dependency  of  each  part  of  the  development 
cycle.  In  other  words,  a  conscious  effort  is 
made  to  present  a  solution  that  lowers  the  to¬ 
tal  risk  for  the  entire  program  -  not  just  the 
short  term  problem.  NAVAIR  frequently  uses 
the  templates  in  the  RFP  process  by  requesting 
contractors  to  provide  information  on  the  tem¬ 
plates  believed  to  be  applicable  to  their  pro¬ 
gram. 

5.5.4  Inputs  and  Outputs 

Since  the  technique  is  not  a  model,  it 
requires  no  formal  inputs.  What  it  does  require 
is  discipline.  Some  amount  of  time  must  be 
spent  in  reading  the  manual  and  using  it  to  ex¬ 
amine  risk  within  a  given  program.  A  practical 
output  of  the  technique  was  the  watchlist  which 
was  described  in  Section  4.4.2. 

5.5.5  Major  Steps  in  Applying 
the  Technique 

Since  the  templates  cover  areas  com¬ 
mon  in  nearly  every  program  it  is  suggested 
that  each  template  be  utilized.  After  reading 
the  material,  individuals  and/or  groups  should 
evaluate  themselves  in  relationship  to  the  solu¬ 
tions/risk  mitigating  actions  suggested  in  the 
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Figure  5.5-1  Critical  Path  Templates 
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template.  For  those  areas  that  are  potential 
“show  stoppers”,  a  separate  watchlist  should  be 
developed  and  maintained.  A  semi-annual  re¬ 
view  of  all  templates  is  recommended  with  up¬ 
dates  as  the  program  progresses. 

5.5.6  Use  of  Results 

The  results  from  the  transition  tem¬ 
plates  can  be  used  in  several  ways:  1)  They  can 
be  used  in  presentations  to  higher  levels  of 
authority;  2)  They  can  be  used  to  influence  the 
contractors  current  level  of  activity  in  an  area; 
3)  They  can  be  used  for  continued  monitoring 
of  progress  in  each  element. 

5.5.7  Resource  Requirements 

Generally,  the  templates  require  that 
the  program  manager  be  involved  in  the  risk 
identification  process.  Inputs  should  be  pro¬ 
vided  by  all  functional  managers.  The  use  of  the 
templates  is  not  intended  to  require  substantial 
special  skills  or  extra  resources. 

5.5.8  Reliability 

Two  cautions  are  applicable  when  us¬ 
ing  this  technique: 

•  Do  not  assume  that  the 
templates  contain  all  possi¬ 
ble  technical  risks  within  a 
given  area.  While  the  com¬ 
mon  problems  are  identi¬ 
fied,  this  is  not  an 
exhaustive  list. 

•  The  templates  do  not  con¬ 
tain  information  regarding 
several  of  the  program¬ 
matic  risk  areas  that 
should  also  be  examined 
for  risk. 


5.6 

5.6.1  General 

Decision  analysis  can  be  used  to  deter¬ 
mine  optional  strategies  when  a  decision 
maker  is  faced  with  several  decision  alterna¬ 
tives  and  an  uncertain  or  risk  filled  pattern  of 
future  events.  Before  selecting  a  specific  deci¬ 
sion  analysis  technique,  the  type  of  decision¬ 
making  situation  that  will  be  encountered  must 
be  considered.  The  classification  method  for 
decision-making  situations  is  based  upon  the 
knowledge  the  decision  maker  has  about  those 
future  events  which  are  beyond  the  decision 
maker’s  control  (known  as  states  of  nature). 
With  this  in  mind,  there  are  two  types  of  deci¬ 
sion-making  situations. 

1)  Decision-making  under 
certainty  -  The  process  of 
choosing  a  decision  alterna¬ 
tive  when  the  states  of  na¬ 
ture  are  known. 

2)  Decision-making  under 
uncertainty  -  The  process 
of  choosing  a  decision  al¬ 
ternative  where  the  states 
of  nature  are  unknown. 

The  decision  analysis  techniques  appro¬ 
priate  for  risk  assessment  are  those  which  take 
into  consideration  that  the  decisions  are  made 
under  uncertainty. 

In  many  situations  where  good  prob¬ 
ability  estimates  can  be  developed  for  the 
states  of  nature,  the  Expected  Monetary  Value 
(EM  V)  method  is  a  popular  technique  for  mak¬ 
ing  decisions.  In  some  situations  of  decision¬ 
making  under  uncertainty,  the  decision-maker 
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may  have  very  little  confidence  in  his  or  her 
ability  to  assess  the  probabilities  of  the  various 
states  of  nature.  In  such  cases,  the  decision¬ 
maker  might  prefer  to  choose  a  decision  crite¬ 
rion  that  does  not  require  any  knowledge  of 
the  probabilities  of  the  states  of  nature. 

5.6.2  Description  of  Technique 

In  general,  there  are  three  steps  in  for¬ 
mulating  a  decision  theory  problem  using  the 
EMV  method. 

1)  The  initial  step  in  the  deci¬ 
sion  theory  approach  is  the 
definition  of  the  problem. 

2)  For  a  given  problem  situ¬ 
ation,  identify  the  alterna¬ 
tives  that  may  be  considered 
by  the  decision-maker.  The 
alternatives  which  are  feasi¬ 
ble  to  the  decision  maker 
may  be  denoted  by  di. 

3)  Identify  those  relevant  fu¬ 
ture  events  which  might 
occur  and  are  beyond  the 
control  of  the  decision¬ 
maker.  These  are  re¬ 
ferred  to  as  states  of 
nature  and  may  be  de¬ 
noted  by  sj . 


In  decision  theory  terminology,  a  par¬ 
ticular  outcome  resulting  from  a  certain  deci¬ 
sion  and  the  occurrence  of  a  particular  state  of 

nature  is  referred  to  as  the  payoff.  V(di ,  sj ) 
denotes  the  payoff  associated  with  decision  al¬ 
ternative  di  and  state  of  nature  sj . 

5.6.3  When  Applicable 

The  EMV  model  is  applicable  during 
any  phase  of  a  program,  although  it  would 


typically  be  generated  at  the  onset  of  the  pro¬ 
gram  to  identify  the  probabilistic  courses  of 
action  the  program  may  take.  Since  decision 
analysis  models  can  be  portrayed  as  decision 
trees  (Figure  5.6-1),  it  can  be  applied  to  net¬ 
work  analysis.  Probabilistic  branching  in  a  net¬ 
work  is  an  example  of  using  decision  analysis 
in  a  network  analysis  framework. 

5.6.4  Inputs  and  Outputs 

The  inputs  to  the  EMV  model  consist 
of  the  decision  alternatives  to  be  considered, 
the  states  of  nature  associated  with  the  decision 
alternatives  and  the  probability  of  occurrence 
for  each  state  of  nature.  The  outputs  of  the 
EMV  method  are  the  expected  monetary  val¬ 
ues  for  each  of  the  decision  alternatives  under 
consideration. 

5.6.5  Mqjor  Steps  in  Applying 
the  Technique 

The  Expected  Monetary  Value  (EMV) 
criterion  requires  that  the  analyst  compute  the 
expected  value  for  each  alternative  and  then 
select  the  alternative  yielding  the  best  ex¬ 
pected  value,  let: 

Let:  P(s j )  —  probability  of  occurrence  for 

the  state  of  nature  sj 

N  =  number  of  possible  states 
of  nature. 

Since  one  and  only  one  of  the  N  states 
of  nature  can  occur,  provided  the  analyst  pro¬ 
vides  disjoint  options,  the  associated  prob¬ 
abilities  must  satisfy  the  following  conditions: 
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n 


P(Sj)  _>  0  for  all  states  of 
1  nature j 

P(sj)  =  P(si  )  + 

JP(s2)  +  ...  +  P(^)  =  1 


The  expected  monetary  value  of  a  decision  al¬ 
ternative  di  is  given  by: 

EMV  (dj)  =  2  P(s  j)  V  (d  j,  Sj) 

j  =  1 


In  other  words,  the  expected  monetary 
value  of  a  decision  alternative  is  the  sum  of  the 
product  of  the  payoffs  with  their  respective 
probabilities-  The  percentage  value  for  a  pay¬ 
off  is  the  probability  of  the  associated  state  of 
nature  and  therefore,  the  probability  the  payoff 
occurs.  The  following  is  an  example  situation  in 
which  the  EMV  model  can  be  used  to  make  a 
decision. 


SAMPLE  PROBLEM 

Consider  the  following  example  of 
whether  or  not  to  conduct  100%  final  system 
tests  on  a  ground  based  radar  system  for  which 
production  has  been  reinstated  for  a  quantity 
of  500.  Historically,  the  radars  failure  rate, 
once  in  the  field,  has  been  4%.  The  cost  to  sub¬ 
ject  each  radar  to  the  required  tests  is  $10,000 
per  radar  (total  cost  =  $5  million).  Also,  the 
nature  of  the  tests  are  such  that  each  radar 
tested  will  have  to  undergo  some  degree  of  re¬ 


work.  Historically,  the  cost  to  reassemble/rein¬ 
stall  each  radar,  that  passes  the  tests,  has 
averaged  $2,000.  However,  the  cost  to  repair  a 
radar  which  has  failed  acceptance  tests  is 
$23,000.  Once  in  the  field,  however,  the  total 
cost  associated  with  repairing  a  defective  radar 
system  escalates  to  $350,000  per  radar.  With 
this  scenario  in  mind,  the  question  is  whether 
or  not  it  is  more  cost  effective  to  conduct  100% 
testing  on  the  radars  or  to  accept  4%  failures  in 
the  field. 

A  decision  table  can  be  constructed 
which  will  portray  this  problem  with  respect  to 
two  decision  alternatives  and  the  respective 
states  of  nature.  Table  5.6-1  depicts  the  deci¬ 
sion  table  for  this  problem  and  the  associated 
analysis. 

From  the  decision  table,  the  analyst  can 
depict  the  problem  in  the  form  of  a  decision 
tree,  which  completely  portrays  the  decision 
(See  Figure  5.6-1).  Although  the  tree  itself  may 
never  be  drawn,  all  relevant  events  must  be 
listed  and  an  analysis  made  to  determine  prob¬ 
lems  that  can  occur  during  each  phase  of  the 
process  of  arrival  at  the  decision  points.  Ex¬ 
perts  are  consulted  to  identify  each  problem 
and  possible  problem  resolutions  which  can  be 
considered  and  to  assign  probabilities  to  the 
various  problems  and  resolutions.  Any  realistic 
and  convenient  number  of  sequential  resolu¬ 
tion  efforts  can  be  postulated. 
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STATES  OF 


PAYOFF 


500  Radars 


,, *  $960,000 


$5  MILLION  +  $960,000 
+  $460,000  =  $6.42  MILLION 


$460,000 


$7  MILLION 


$7  MILLION 


Figure  5.6-1  Decision  Tree 


Table  5.6-1  Decision  Table 


DECISION  ALTERNATIVES 


FAIL 

P(Si  )  =  .04 


PASS 

P(S2  )  *  .96 


TEST  EACH  RADAR  dt  =  $5,000,000 


500  Radars  (.04  failures) 
($23K  rework/radars) 


500  Radars  (.96  pass) 
($2000  rework/radars) 


NO  TEST  d2  =  0 


.04  failures(5Q0  radars) 
($350K/radars) 


Analysis: 

EMV  (test)  =  500  radars  ($10,000  test/radar)  +  500  radars  (.04  failures)  ($23K  rework/radar) 
+  500  radars  (.96  pass)  ($2,000  rework/radar)  =  $6.42  million 

EMV  (no  test)  =  .04  failures  in  field  (500  radars)  ($350, 000/radar)  =  $7  million 


Since  objective  is  to  minimize  cost,  decision  would  be  to  test  radars. 
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5.6.6  Use  of  Results 

Given  the  expected  monetary  values  of 
the  decision  alternatives,  the  analyst’s  selection 
of  the  appropriate  alternative  is  predicated  on 
whether  the  objective  is  to  maximize  profit  or 
to  minimize  cost.  For  the  sample  problem, 
since  the  objective  was  to  minimize  cost,  the 
analyst  would  select  the  alternative  with  the 
lowest  EMV.  When  the  difference  between 
one  or  more  decision  alternatives  is  small, 
other  programmatic  factors  may  be  taken  into 
consideration  when  making  the  decision. 

5.6.7  Resource  Requirements 

With  respect  to  resource  requirements, 
the  EMV  technique  is  simplistic  and  can  usu¬ 
ally  be  easily  calculated  once  the  inputs  to  the 
model  have  been  obtained.  As  the  decision 
problem  being  modeled  becomes  more  com¬ 
plex,  with  an  increasing  number  of  decision  al¬ 
ternatives  and  states  of  nature,  the  time 
required  to  create  a  decision  table  or  a  decision 
tree  will  also  increase. 

5.6.8  Reliability  of  Results 

One  of  the  most  attractive  features  of 
the  EMV  method  of  decision  analysis  is  that 
once  the  respective  inputs  to  the  model  have 
been  obtained,  there  is  no  ambiguity  insofar  as 
the  analysis  is  concerned.  The  reliability  of  the 
results  are  predicated  on  the  validity  of  the  in¬ 
puts  to  the  model;  that  is,  with  what  degree  of 
accuracy  the  analyst/experts  can  define  all  the 
relevant  decision  alternatives,  states  of  nature, 
and  respective  probabilities. 


Another  significant  benefit  of  the  EMV 
method  is  that  it  diagrammatically  portrays  the 
decision  alternatives  and  the  associated  analy¬ 
sis,  making  it  easier  to  conceptually  understand 
the  problem,  the  alternatives,  and  the  analysis. 

5.7  ESTIMATING  RELATIONSHIP 

5.7.1  General 

The  estimating  relationship  method  en¬ 
ables  program  office  personnel  to  evaluate  a 
program,  and  based  thereon,  use  an  equation 
to  determine  an  appropriate  management  re¬ 
serve  or  risk  funds  budget.  When  using  this 
method,  the  management  reserve  funds  repre¬ 
sent  the  amount  of  funding,  over  and  above 
that  determined  by  cost  analysis  alone,  re¬ 
quired  for  work  associated  with  unanticipated 
risks.  This  method  was  originally  developed 
and  is  still  used  for  contract,  not  program 
costs.  The  management  reserve  funds  require¬ 
ment  computed  is  usually  expressed  as  a  per¬ 
centage  of  the  baseline  cost  estimate.  The 
technique  is  called  an  estimating  relationship 
method  because  it  uses  some  of  the  same  tech¬ 
niques  associated  with  cost  estimating  rela¬ 
tionships  (CERs),  used  in  parametric  cost 
estimating. 

5.7.2  Description  of  Technique 

The  cost  estimating  relationship 
method  is  based  on  the  observation  that  costs 
of  systems  seem  to  correlate  with  design  or 
performance  variables.  The  independent  vari- 
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ables,  often  called  explanatory  variables,  are 
analyzed  using  regression  analysis  to  describe 
the  underlying  mechanism  relating  such  vari¬ 
ables  to  cost.  This  approach  to  cost  estimating, 
also  called  parametric  cost  estimating,  is  widely 
accepted  and,  even  for  complex  functions,  is 
easy  to  apply. 

This  ease  of  application  makes  it  natu¬ 
ral  to  attempt  to  use  the  same  techniques  to  es¬ 
timate  the  costs  resulting  from  risks.  The 
approach  attempts  t  j  discover  acquisition  pro¬ 
gram  or  contract  characteristics,  as  explanatory 
variables,  which  can  then  be  correlated  with 
the  historically  demonstrated  need  for  man¬ 
agement  reserve  or  risk  funds.  Regression 
analysis  using  “actual”  management  reserve 
funds  from  past  programs,  expressed  as  a  per¬ 
cent  of  total  costs,  is  performed  to  develop  an 
equation  with  which  to  estimate  management 
reserve  fund  requirements  for  a  new  program, 
not  in  a  database. 

The  application  of  this  technique  is  de¬ 
scribed  in  Section  5.7.5.  In  the  example  de¬ 
scribing  the  application  of  this  technique,  four 
program  and  prime  contractor  characteristics, 
which  are  known  to  affect  the  level  of  uncer¬ 
tainty,  are  evaluated  by  PMO  personnel.  Each 
characteristic  is  assigned  a  value  based  on  a  dif¬ 
ferent  scale  provided  for  each  characteristic. 
The  four  characteristics  used  are  Engineering 
Complexity  (zero  to  five),  Contractor  Profi¬ 
ciency  I  Experience  (zero  to  three),  Degree  of  Sys¬ 
tem  Definition  (zero  to  three),  and  Multiple 
Users  (zero  or  one).  The  sum  of  these  numerics 
is  entered  as  the  value  X,  in  an  estimating 
equation  such  as  Equation  5.7-1. 


y  =  (0.192  -  0.037  X+  .009  X2)  x  100 

Equation  5.7-1  (Ref  5- 1 ) 

This  formula  determines  the  percentage  man¬ 
agement  reserve  requirement,  y.  The  particular 
model  shown  in  this  example  is  usable  only  for 
X  values  between  2  and  10.  Lower  values  indi¬ 
cate  essentially  no  need  for  management  re¬ 
serve  funds. 

5.73  When  Applicable 

This  method  of  estimating  the  addi¬ 
tional  funding  needed  to  cover  anticipated 
risks  has  limited  application.  First  it  can  only  be 
used  if  the  research  has  already  been  done  to 
establish  a  valid  historical  relationship  be¬ 
tween  the  key  program  or  contract  characteris¬ 
tics  of  similar  programs,  and  management 
reserve  funding  requirements.  This  was  done 
at  the  USAF  Electronics  Systems  Division 
(Ref.  5-1).  However,  no  other  DoD  users  of 
this  type  of  method  were  found  during  prepa¬ 
ration  of  the  risk  handbook.  The  method  is 
most  applicable  in  the  circumstances  where 
good  historical  program  description  and  man¬ 
agement  reserve  funding  requirements  are 
available  for  several  similar  programs.  If  the 
required  risk  funding  estimating  relationship  is 
available,  this  method  has  the  advantage  that  it 
is  both  quick  and  easy  to  apply. 

5.7.4  Inputs  and  Outputs 

Input  -  The  inputs  to  an  estimating  rela¬ 
tionship  model,  such  as  Eq.  5.7-1  are  judg¬ 
ment  values  characterizing  the  four  program  or 
contract  factors  described  in  Section  5.7.2. 
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Output  -  The  estimating  relationship 
method  provides  a  percentage  figure  to  be  ap¬ 
plied  to  estimated  baseline  cost  to  be  used  to 
determine  the  amount  of  total  or  contract  man¬ 
agement  reserve  funds  required.  This  percent¬ 
age  value  is  computed  using  an  equation  like 
Eq.  5.7-1,  with  the  X  value  being  the  sum  of 
the  four  factor  values  determined  by  PMO  per¬ 
sonnel. 

5.7.5  Mqjor  Steps  In  Applying 
The  Technique 

Assuming  an  appropriate  management 
reserve  estimating  equation  is  not  available, 
the  first  major  step  in  using  this  method  and  by 
far  the  most  difficult,  is  developing  an  equation 
relating  program  characteristics  to  manage¬ 
ment  reserve  funding  requirements.  The  most 
difficult  part  of  this  step  is  finding  valid  histori¬ 
cal  characteristic  and  management  reserve 
funding  data  for  enough  similar  programs  to 
carryout  regression  analysis.  Data  from  at  least 
ten  past  programs  should  be  used  to  develop  an 
estimating  relationship  equation. 

The  second  part  of  Step  1  is  to  deter¬ 
mine  the  program  or  contract  characteristics 
which  drive  management  reserve  funding  re¬ 
quirements,  and  for  which  historical  data  has 
been  collected.  After  the  historical  data  has 
been  collected,  it  is  relatively  simple  to  use  re¬ 
gression  analysis  to  identify  these  characteris¬ 
tics.  The  summing  of  judgment  level  values  for 
each  of  four  program  characteristics  as  done 
by  Electronic  Systems  Division  (ESD)and  de¬ 


scribed  in  Section  5.7.2,  is  only  one  way  to  de¬ 
velop  one  or  more  independent  variables  for 
an  estimating  relationship  for  management  re¬ 
serve  funding  requirements.  Geometric  mean 
or  weighted  average  techniques  could  also  be 
used.  Multiple  regression  analysis  techniques 
frequently  are  used  for  parametric  cost  esti¬ 
mating.  The  second  and  final  major  step  in  us¬ 
ing  this  method  is  to  use  the  prediction 
equation  derived  through  regression  analysis 
uad  the  new  program  or  contract  characteristic 
information  to  compute  a  percent  value  for  the 
additional  management  reserve  funds  needed 
to  cover  anticipated  additional  costs  associated 
with  risk.  It  may  be  useful  to  vary  the  program 
description  characteristic  data  somewhat  and 
recompute  the  estimating  equation  to  assess 
the  impact  of  such  changes  in  the  computed 
management  reserve  requirements.  This  sensi¬ 
tivity  analysis  is  usually  prudent  because  of  the 
uncertainty  associated  with  the  predicted  pro¬ 
gram  or  contract  characteristics. 

5.7.6  Use  of  Results 

Using  this  method,  the  percent  value  of 
the  estimated  contract  or  program  cost  is  com¬ 
puted  and  added  to  the  basic  cost  estimate  to 
cover  funds  needed  for  risk.  As  an  example,  if 
the  contract  cost  estimate  was  S100M  and  the 
prediction  equation  provided  a  result  of  30 
percent,  $30  million  dollars  would  be  added  for 
risk,  making  the  total  estimated  contract  cost 
$130M. 
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5.7.7  Resource  Requirements 

Once  a  suitable  management  reserve 
funding  requirement  prediction  equation  is 
available,  only  a  few  hours  are  required  to  ap¬ 
ply  this  method.  Most  of  the  effort  required  in¬ 
volves  interviewing  PMO  personnel  to  obtain 
their  judgments  on  the  contract  or  program 
characteristic  values  to  be  used.  If  a  prediction 
equation  has  to  be  developed,  one  to  three 
months  of  a  skilled  analyst’s  time  would  be  re¬ 
quired,  depending  on  the  difficulty  incurred  in 
acquiring  the  needed  data.  It  is  possible  that 
the  required  data  may  not  be  available,  and 
that  any  amount  of  effort  would  not  result  in 
the  development  of  a  satisfactory  prediction 
equation. 

5.7.8  Reliability 

This  method  as  implemented  by  the 
ESD  model  provides  results  that  significantly 
increase  cost  estimates  (based  primarily  on  the 
extrapolation  of  historical  data  which  may  in¬ 
clude  costs  for  risks  that  have  already  been  ex¬ 
perienced)  to  allow  for  risk.  Because  the 
additional  funds  are  based  primarily  on  judg¬ 
ment  values,  they  are  subject  to  question.  If 
this  technique  is  to  be  used,  it  would  always  be 
prudent  for  a  PMO  to  have  the  method  in¬ 
cluding  the  prediction  equation  to  be  used,  re¬ 
viewed  and  accepted  by  higher  headquarters, 
before  using  it  as  the  basis  for  a  sizable  request 
for  additional  funds  to  cover  risks.  The  method 
can  only  be  used  where  adequate  historical 
data  is  available  to  develop  a  sound  manage¬ 
ment  reserve  fund  requirement  prediction 
equation. 


5.8 

5.8.1  General 

Many  program  managers  are  familiar 
with  the  concept  of  network  based  scheduling 
as  a  program  management  tool.  Program  man¬ 
agers  are  fully  aware  that  a  quality  schedule  is 
critical  for  the  effective  planning,  implement¬ 
ing,  and  controlling  of  any  program.  A  quality 
schedule  is  essentially  a  plan  of  action  that  is 
objective  oriented.  It  includes  activities/events 
which  must  be  accomplished  to  achieve  the  de¬ 
sired  objective.  Network  based  scheduling  or 
networking  formalizes  the  scheduling  process 
and  results  in  a  graphical  output  which  displays 
not  only  the  activities  which  must  be  accom¬ 
plished  to  complete  the  program,  but  also  the 
relationships  among  the  activities  (that  is. 
which  activities  precede,  succeed,  or  are  paral¬ 
lel  to  other  activities).  The  utility  of  networking 
in  general  includes: 

•  Focusing  the  attention  of 
all  management  levels  dur¬ 
ing  the  planning  phase 

•  Estimating  program  com¬ 
pletion  date 

•  Displaying  the  scope  of 
the  program 

•  Assessing  resource  re¬ 
quirements 

•  Facilitating  “what  if”  exer¬ 
cises 

•  Highlighting  critical  activi¬ 
ties 

•  Evaluating  performance. 

The  keys  for  successful  network  development 
are: 
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•  Determine  the  appropriate 
level  of  detail  (aggregate, 
intermediate,  detailed) 

•  Identify  relevant  activities 

•  Define  relationships 
among  activities  (Depend¬ 
ency,  concurrency) 

•  Forecast  time  durations 

•  Involvement  of  all  rele¬ 
vant  individuals  in  all  of 
the  above. 

In  many  situations  program  managers 
assume  the  responsibility  for  planning,  sched¬ 
uling,  and  controlling  projects  that  consist  of 
numerous  separate  jobs  or  tasks  performed  by 
a  variety  of  departments,  program  offices,  indi¬ 
viduals,  etc.  Often,  these  programs  are  so  com¬ 
plex  and/or  large  that  the  program  manager 
cannot  possibly  remember  all  the  information 
pertaining  to  the  plan,  schedule,  and  progress 
of  the  program. 

In  these  situations  the  techniques  of 
PERT  (Program  Evaluation  and  Review  Tech¬ 
nique)  and  CPM  (Critical  Path  Method)  have 
proven  to  be  extremely  valuable  in  assisting 


program  managers  in  carrying  out  their  pro¬ 
gram  management  responsibilities.  Besides 
being  one  of  the  original  scheduling  tech¬ 
niques,  PERT,  which  was  developed  during 
the  Polaris  submarine  program  in  the  late 
1950’s  was  also  the  first  risk  analysis  tool.  The 
objectives  of  PERT  were  to  manage  schedule 
risk  by  establishing  the  shortest  development 
schedule,  to  monitor  project  progress,  and  to 
fund  or  apply  necessary  resources  for  maintain¬ 
ing  the  schedule.  Figure  5.8-1  represents  a 
PERT  network. 

One  of  the  most  significant  outputs  of  a 
network  is  the  identification  of  the  critical  path. 
The  critical  path  consists  of  those  program  ac¬ 
tivities  which  must  be  completed  on  schedule 
or  the  overall  program  completion  date  will 
slip.  Activities  on  the  critical  path  are  the  “long 
poles  in  the  tent”.  In  addition,  activities  can  be 
assigned  unique  identifier  codes.  One  of  the 
many  options  this  permits  is  the  capability  to 
select  those  activities  related  to  a  specific  WBS 
element  which  are  on  the  critical  path.  Activi¬ 
ties  which  are  not  on  the  critical  path  have 


Figure  5.8-1  Program  Represented  as  a  Network 
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slack  time  associated  with  them.  This  means 
that  there  is  some  amount  of  time  that  the  ac¬ 
tivity’s  scheduled  completion  date  can  slip 
without  impacting  the  overall  program  com¬ 
pletion  date. 

5.8.2  Description  of  Technique 

The  original  networking  technique  was 
based  on  the  Arrow  Diagram  Method  (ADM) 
or  “activity  on  arrow”  method  of  representing 
the  logical  relationships  between  activities. 
ADM  represents  all  predecessor  and  successor 
activities  as  finish  to  start  relationships.  Succes¬ 
sor  activities  are  not  initiated  until  the  prede¬ 
cessor  is  100%  complete.  However,  since  this 
form  of  relationship  is  not  always  true  for 
predecessor/successor  activities,  other  net¬ 
working  methodologies  were  developed  to 
more  accurately  reflect  the  realities  of  prede¬ 
cessor/successor  dependencies.  Newer  com¬ 
puter-based  networking  systems  use  the 
Precedence  Diagramming  Method  (PDM)  or 
“activity  on  node”  to  represent  network  logic. 
PDM  allows  greater  flexibility  than  ADM  in 
describing  predecessor/successor  relation¬ 
ships.  With  PDM,  the  following  relationships 
can  be  described  in  addition  to  finish  to  start: 

Finish  to  Finish  -  successor 
activity  cannot  finish  until 
some  user  specified  period 
of  time  after  the  predeces¬ 
sor  has  completed. 

Start  to  Start  -  the  succes¬ 
sor  activity  cannot  start 
until  some  user  specified 
period  of  time  after  the 
start  of  the  predecessor. 

Start  to  Finish  -  the 
predecessor  activity  can¬ 


not  be  completed  until 
some  specified  period  of 
time  after  the  predecessor 
has  started. 

Newer  network  based  risk  models  use 
PDM  as  well  as  conventional  ADM.  The  de¬ 
scription  that  follows  is  based  on  the  traditional 
ADM  networks  because,  to  date,  they  are  more 
popular  as  risk  tools.  PDM  however,  is  more 
popular  as  a  scheduling  tool. 

To  accurately  reflect  the  realities  of  risk 
related  issues,  the  PERT  method  of  network 
analysis  has  been  enhanced  over  the  years. 
Logic  has  been  added  which  increases  the  func¬ 
tionality  of  network  analysis  as  a  risk  analysis 
tool.  Because  of  the  changes,  some  of  the  old 
terminology  has  been  replaced.  The  lines  are 
known  as  arcs  instead  of  activities.  Decision 
points  at  the  initiation  or  completion  of  activi¬ 
ties  and  milestones  are  referred  to  as  “nodes". 
Nodes  can  be  of  three  types: 

1)  Source  nodes  -  indicate  the 
initiation  of  the  program. 

2)  Intermediate  nodes  -  indi¬ 
cate  milestones  or  the  in¬ 
itiation  and  termination  of 
arcs. 

3)  Terminal  nodes  -  repre¬ 
sent  the  completion  of 
the  program  or  the  fail¬ 
ure  to  complete  some 
segment  of  the  program. 

In  a  probabilistic  network,  there  are  two 
ways  in  which  uncertainty  manifests  itself. 
First,  activities  may  have  uncertain  outcomes  in 
terms  of  time  to  complete,  cost  to  complete,  or 
achievement  of  some  technical  level  of  per- 
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formance.  Generally,  technical  performance  is 
held  as  a  fixed  parameter  while  the  other  two 
vary.  Second,  the  initiation  of  activities  ema¬ 
nating  from  a  node  may  be  predictable  only  in 
a  probabilistic  way.  For  example,  a  test  out¬ 
come  (pass/fail)  may  determine  whether  the 
next  activity  is  a  progressive  continuation  of  a 
plan  or  a  corrective  action.  Since  the  test  out¬ 
come  cannot  be  predicted  with  certainty,  it  as¬ 
sumes  a  probabilistic  nature.  The  network 
model  represents  this  by  showing  at  least  two 
arcs  emanating  from  the  node  representing  test 
activity  completion.  The  analyst  can  assign 
probability  functions  to  the  arcs  to  represent 
the  relevant  probabilities  of  completing  within 
time  or  cost  constraints  or  of  meeting  perform¬ 
ance  levels. 

An  important  aspect  of  network  models 
that  is  needed  to  permit  realistic  simulation  of 
programs  is  varied  “node  logic”.  Node  logic  re¬ 
fers  to  the  rules  which  determine  when,  for  ex¬ 
ample,  a  decision  point  is  passed  and  when  a 
subsequent  activity  initiates. 

The  more  advanced  computer  pro¬ 
grams  will  allow  use  of  both  “AND”  and 
“OR”  logic  and  “DETERMINISTIC”  and 
“PROBABILISTIC”  output  node  logic.  The 
two  types  of  input  logic  determine  whether  all 
(“AND”  logic),  only  one  (exclusive  “OR” 
logic),  or  some  (“OR”  logic)  of  the  possible 
arcs  entering  a  node  must  be  completed  for 
the  node  to  be  actuated.  The  two  output  logics 
determine  whether  all  (“DETERMINISTIC” 
logic)  or  only  one  (“PROBABILISTIC”  logic) 
arc  is  initiated  upon  completion  of  node  actua¬ 
tion. 


As  previously  mentioned,  of  fundamen¬ 
tal  importance  for  network  development  is  the 
selection  of  the  appropriate  level  of  network 
detail.  The  consensus  is  that  completion  of  a 
high  aggregate  level  of  detail  should  be  accom¬ 
plished  before  attempting  to  model  the  details 
of  the  program  structure.  Aggregate  level  net¬ 
works  will  provide  a  more  realistic  determina¬ 
tion  of  what  the  detail  level  networks  will 
contain.  However,  aggregate  level  networks 
will  also  contain  more  inherent  uncertainty 
than  would  be  the  case  at  a  finer  level  of  detail. 
As  the  program  requirements  and  information 
become  more  readily  available,  the  network 
models  will  evolve  to  a  greater  level  of  detail. 

5.8.3  When  Applicable 

Network  analysis  has  universal  applica¬ 
tion  in  the  program  offices.  Networks  are  for¬ 
mulated  based  on  program  activities, 
interrelationships  among  activities,  and  con¬ 
straints  (time,  money,  manpower,  technology, 
etc).  Because  all  programs  have  these  charac¬ 
teristics,  network  analysis  is  universally  appli¬ 
cable.  The  application  of  network  analysis  is 
made  easier  if  network  based  program  sched¬ 
ules  already  exist.  If  this  is  the  case,  the  analyst 
can  make  the  logic  modifications  required  so 
that  the  network  information  can  be  readily  in¬ 
put  into  a  risk  analysis  software  program.  If  a 
network  does  not  already  exist,  one  must  be 
created.  The  time  savings  which  can  be  in¬ 
curred  transforming  an  existing  network  versus 
creating  one  provides  a  strong  argument  in  fa¬ 
vor  of  network  based  program  scheduling  from 
the  onset  of  a  program. 
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5.8.4  Inputs  and  Outputs 

The  input  for  the  development  of  the 
network  risk  model  consists  of  probability  den¬ 
sity  functions  (See  Section  5.2  and  Appendix  F 
for  discussion  on  some  of  the  techniques  avail¬ 
able  for  quantifying  expert  judgment).  Since  in¬ 
put  to  the  network  model  may  initially  be 
qualitative  judgment  which  must  be  trans¬ 
formed  into  quantitative  information,  it  is  im¬ 
perative  that  all  individuals  who  play  a  relevant 
programmatic  role  provide  input  during  the 
development  process.  The  credibility  of  the  re¬ 
sulting  network  is  affected  by  the  extent  to 
which  knowledgeable,  relevant  program  per¬ 
sonnel  contribute  to  its  development.  Standard 
output  from  network  risk  models  includes 
probability  curves,  barcharts  comparing 
baseline  and  “risk  free”  schedules,  cost  histo¬ 
grams,  Cumulative  Density  Functions  (CDFs), 
the  mean,  standard  deviation  of  the  sample, 
coefficient  of  variation,  and  mode  for  all  speci¬ 
fied  decision  points  and  activities.  These  re¬ 
sult  from  executing  a  Monte  Carlo  simulation 
of  the  network.  This  is  simply  modeling  the  exe¬ 
cution  of  the  program  many  times. 

Most  packages  also  produce  a  “critica¬ 
lity  index”  for  each  activity.  This  index  shows 
how  often  each  activity  appeared  on  the  critical 
path  during  the  course  of  the  simulation  proc¬ 
ess.  Cost  curves  and  histograms  can  also  be 
produced  which  may  indicate  the  direction  the 
project  is  taking.  This  information  can  be  used 
to  continually  adjust  labor,  material,  and  time 
estimates. 


5.8.5  Steps  in  Applying  the 
Technique 

The  first  step  in  the  process  of  modeling 
a  program  is  for  the  analyst/manager  to  manu¬ 
ally  develop  a  rough-cut  network.  In  order  to 
develop  a  realistic  model  of  the  program,  it  is 
crucial  that  the  analyst  identify  all  the  relevant 
parameters  such  as  nodes,  arcs,  logic  relation¬ 
ships,  and  Probability  Density  Functions 
(PDFs).  As  previously  stated,  all  relevant  pro¬ 
gram  personnel  should  play  a  role  in  develop¬ 
ing  and  validating  the  network. 

Once  the  rough-cut  network  has  been 
developed,  the  analyst  can  input  the  informa¬ 
tion  into  the  computer  for  processing.  There 
are  many  software  packages  currently  avail¬ 
able  for  network  risk  analysis.  The  whole  spec¬ 
trum  from  mainframe  to  microcomputer 
applications  is  covered  by  available  software. 
Some  of  the  packages  currently  available  in¬ 
clude: 

•  PROSIM 

•  VERT 

•  VERTPC 

•  RISNET 

•  PROJECT/2 

•  OPERA 

and  others. 

Once  the  iterative  process  of  develop¬ 
ing  the  rough-cut  network  has  been  com¬ 
pleted,  the  data  is  ready  for  input  and 
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processing  by  the  computer.  Using  the  process 
known  as  Monte  Carlo  simulation,  the  soft¬ 
ware  determines  the  most  likely  course  of 
events.  Since  one  simulation  conveys  little  use¬ 
ful  information,  the  Monte  Carlo  simulation 
repeats  the  process,  recalculating  the  critical 
path,  as  many  times  as  necessary  (or  as  defined 
by  the  user)  to  account  for  all  possible  scenar¬ 
ios.  Typically,  1,000  to  6,000  simulations  are 
processed.  The  result  of  these  simulations  is  a 
statistically  calculated  scenario  that  predicts 
the  eventual  course  of  the  project  with  a  confi¬ 
dence  le^el  as  specified  by  the  user. 

5.8.6  Use  of  Results 

The  output  of  the  network  risk  analysis 
process  is  extremely  useful  to  the  program 
manager.  The  performance  of  network  risk 
analysis  generally  provides  an  in-depth  under¬ 
standing  of  the  sources  and  degree  of  risks.  The 
results  of  the  risk  analysis  process  provide  the 
information  required  to  effectively  execute  the 
“risk  handling”  phase  of  risk  management. 

5.8.7  Resource  Requirements 

Since  most  network  risk  assessments 
accomplished  in  the  DoD  are  carried  out  by 
functional  support  offices,  risk  assessment  dol¬ 
lar  costs  should  be  estimated  from  manpower 
requirements.  A  comprehensive  network 
analysis  for  a  major  program  may  require  defi¬ 
nition  of  between  200  and  1000  activities  and 
require  two  to  six  man-months  of  GS-12  to 
GS- 1 4  analyst  effort  for  gathering  information 
from  subject  expeits  for  use  ir  formulating 
probability  density  functions  (PDFs)  and  for 


building  the  network.  Obtaining  the  informa¬ 
tion  required  to  construct  the  network  usually 
entails  more  time  and  rechecking  than  might 
initially  seem  necessary.  This  is  because  the 
program  plan  is  usually  under  continual  revi¬ 
sion  and  definition,  and  the  support  personnel 
do  not  fully  understand  the  relationships 
among  the  program  activities. 

Although  the  difficulty  and  time  re¬ 
quired  for  network  definition  can  pose  a  prob¬ 
lem,  the  effort  of  constructing  a  consistent  and 
acceptable  network  model  forces  the  responsi¬ 
ble  participants  to  plan  effectively  and  to  un¬ 
derstand  how  their  own  segment  of  the 
program  fits  into  the  whole.  Program  manag¬ 
ers  have  indicated  that  this  benefit  alone  can 
justify  all  the  effort  for  accomplishment  of  a 
formal  network  risk  assessment/analysis. 

5.8.8  Reliability 

The  reliability  of  network  risk  analysis 
is  a  function  of  multiple  factors.  The  develop¬ 
ment  of  a  network  which  accurately  reflects  the 
activities  and  relationships  among  activities  is 
crucial  to  the  resulting  network  analysis.  This  is 
why  it  is  imperative  that  all  relevant  program 
personnel  provide  input  to  the  development 
and/or  modification  of  the  network.  The  defi¬ 
nition  of  PDFs  for  the  cost,  schedule,  and  per¬ 
formance  aspects  of  the  program  is  also  of 
fundamental  importance.  Since  the  Monte 
Carlo  simul  itions  which  predict  the  course  of 
the  project  are  based  on  the  respective  PDFs, 
the  accuracy  of  the  PDFs  in  describing  the  cost, 
schedule,  and  performance  parameters  of  the 
program  is  critical  for  a  reliable  analysis.  The 
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more  reliable  the  network  developed,  the  more 
reliable  the  analysis  will  be. 

5.9  LIFE  CYCLE  COST  ANALYSIS 

5.9.1  General 

A  survey  of  program  management  of¬ 
fices  indicated  that  directed  funding  cuts  most 
often  were  viewed  as  the  source  of  risk  having  a 
major  impact  on  program  execution.  In  order 
to  control  the  adverse  consequences  of  such  a 
risk,  a  program  manager  needs  to  be  able  to 
quickly  determine  the  potential  cost  implica¬ 
tions  of  new  information  such  as  funding  con¬ 
straints  pertinent  to  the  program.  Other 
information  affecting  program  costs  include 
new  knowledge  about  a  wide  range  of  things 
such  as  test  failures  resulting  in  schedule  slips, 
or  directed  production  rate  reductions.  The 
program  manager  also  needs  to  have  quick  ac¬ 
cess  to  the  potential  cost  implications  of  some 
of  the  choices  that  must  be  made  as  the  uro¬ 
gram  progresses.  Many  programs  meet  such 
needs  with  a  computerized  life  cycle  cost  (LCC) 
model.  These  models  are  sometimes  called 
quick  reaction  cost  models  or  quick  reaction 
models.  Such  models  can  be  useful  for  cost  es¬ 
timating,  tradeoff  analysis,  production  rate  and 
quantity  analysis,  warranty  analysis,  sensitivity 
analysis,  and  logistic  support  studies.  Simpler 
models  such  as  the  Quick  Cost  model  devel¬ 
oped  by  DSMC,  are  focused  specifically  on  the 
cost  implication  of  changes  in  yearly  produc¬ 
tion  quantities. 


5.9.2  Description  of  Technique 

The  Life  Cycle  Cost  technique  consists 
of  a  series  of  equations  which  compute  pro¬ 
gram  costs  based  on  product  and  program  in¬ 
formation.  The  exact  nature  of  such 
information  will  be  addressed  in  Section  5.9.4. 
However,  it  will  vary  from  model  to  model,  and 
may  vary  significantly  from  program  to  pro¬ 
gram,  depending  on  the  nature  of  the  program 
and  its  status.  An  important  aspect  of  life  cy¬ 
cle  cost  models  is  that  given  some  informed 
inputs,  the  model  can  be  run  quickly  and  not 
only  provide  a  new  total  life  cycle  cost  estimate, 
but  also  can  give  some  insight  into  where  the 
costs  are  likely  to  change.  The  model  equations 
are  usually  developed  based  on  logic  and  expe¬ 
rience  on  similar  past  programs.  The  cost  ele¬ 
ments  of  life  cycle  cost  models  vary 
significantly.  However,  where  applicable,  they 
usually  include  development,  production,  and 
the  full  spectrum  of  extended  operating  and 
support  costs. 

5.9.3  When  Applicable 

Use  of  a  life  cycle  cost  model  is  appli¬ 
cable  whenever  a  manager  needs  a  quick  esti¬ 
mate  of  the  cost  implications  of  a  past  or 
pending  event.  However,  the  timely  develop¬ 
ment  of  useful  cost  estimates  is  totally  depend¬ 
ent  upon  having  a  completed  and  tested  life 
cycle  cost  model  available  for  immediate  use. 
Such  a  model  is  very  applicable  to  situations 
where  budget  cuts  are  proposed  by  higher 
authority  and  the  PMO  has  only  a  short  time  to 
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describe  the  impact  of  such  cuts.  Program 
managers  can  get  into  trouble  trying  to  buy 
half  the  quantity  for  half  the  cost  or  the  same 
quantity  over  a  longer  period,  for  the  same 
cost. 

5.9.4  Inputs  and  Outputs 

Inputs  -  Most  life  cycle  cost  models 
have  extensive  inputs  that  vary  from  model  to 
model.Timely  use  of  these  models  dictates  that 
input  values  be  continually  maintained  so  only 
those  that  would  change  because  of  recent  or 
pending  actions  need  to  be  obtained  to  carry 
out  the  desired  cost  analysis.  This  is  especially 
important  when  using  detailed  lifecycle  cost 
models  which  aggregate  costs  based  on  the 
characteristics  of  many  individual  subsystems 
and  line  replaceable  units.  Important  input 
values  common  to  many  life  cycle  cost  models 
include: 

•  Production  quantity  by  year 

•  Development  test  quanti¬ 
ties 

•  Cost  quantity  curve  slopes 

•  Support  equipment  re¬ 
quirements 

•  Number  of  bases  to  which 
equipment  will  be  de¬ 
ployed 

•  Spares  requirements 

•  Tooling  costs  and  other 
non-recurring  production 
costs 

•  Deployment  life  of  system 


•  Planned  utilization  rate 
(i.e.,  operating  hours  per 
year) 

•  Failure  rates,  sometimes 
by  subsystem  or  even 
component. 

Outputs  -  As  with  model  inputs,  the  na¬ 
ture  and  format  of  the  outputs  vary  widely 
among  life  cycle  cost  models.  However,  one 
output  option  should  include  an  overall  sum¬ 
mary  of  total  life  cycle  costs  broken  out  only  by 
appropriation  type,  (i.e.,  development,  produc¬ 
tion  and  operation).  Other  useful  output  op¬ 
tions  include  breakouts  of  the  total  life  cycle 
cost  by: 

•  Year 

•  Cost  element 

•  Equipment  component 

•  Combinations  of  the 
above. 

Output  values  may  be  in  fixed  and  speci¬ 
fied  base  year  dollar  values,  or  if  inflation  rates 
were  provided  in  the  input,  as  dollar  values  in¬ 
flated  to  the  year  in  which  they  must  be  appro¬ 
priated. 

5.9.5  Major  Steps  in  Applying 
the  Technique 

The  first  major  step  in  using  a  life  cycle 
cost  model  is  to  develop  a  model  tailored  to 
the  nature  of  the  program  and  anticipated  cost 
information  needs.  This  is  a  key  step  because 
without  it  generation  of  timely  life  cycle  cost 
estimates  will  not  be  possible.  Developing  such 
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a  model  and  gathering  the  required  input  val¬ 
ues  will  usually  require  a  significant  resource 
commitment.  However,  this  effort  can  often  be 
significantly  reduced  by  tailoring  an  existing 
life  cycle  cost  model  already  in  use  for  a  similar 
system. 

The  second  major  step  is  using  the  life 
cycle  cost  model  to  address  a  specific  issue. 
This  could  require  a  data  collection  effort,  but 
it  should  be  significantly  less  than  the  initial  ef¬ 
fort  to  develop  a  model  tailored  to  a  specific 
PMO.  If  a  model  is  already  available  and  pro¬ 
grammed  on  a  computer,  gathering  the  input 
data  required  to  run  the  model  is  almost  always 
the  largest  part  of  the  effort  required  to  pre¬ 
pare  a  life  cycle  cost  estimate. 

The  last  major  step  is  to  review  the 
model  output  and  assure  that  the  results  are 
reasonable  and  address  the  questions  at  issue. 
Any  single  life  cycle  cost  analysis  might  involve 
computation  of  several  to  many  life  cycle  cost 
estimates.  The  life  cycle  cost  model  is  only  a 
very  crude  abstraction  of  the  real  world.  There¬ 
fore,  decision  makers  often  demand  and  will 
always  appreciate  logical  arguments  that  tend 
to  substantiate  the  numerical  results  provided 
by  the  model.  It  is  often  prudent  to  use  the 
model  to  do  sensitivity  analysis  using  a  range 
of  input  values  around  the  primary  input  values 
to  see  how  the  changes  affect  the  model  com¬ 
puted  life  cycle  cost  estimates. 

5.9.6  Use  of  Results 

Life  Cycle  Cost  (LCC)  analysis  results 
can  be  used  to  assess  costs  and,  thereby  cost 


risks  associated  with  many  decision  issues.  Life 
cycle  cost  models  can  be  used  to  develop  or 
carry  out: 

•  LCC  estimates 

•  Production  rate  and  quan¬ 
tity  analyses 

•  Design  trade-off  analysis 

•  Cost  driver  sensitivity 
analyses 

•  Resource  projections  (e.g., 
manpower,  support  equip¬ 
ment) 

•  Repair  level  analyses 

•  Warranty  analyses 

•  Spares  provisioning  and 
optimization 

•  Reliability  growth  analy¬ 
ses 

•  Operational  availability 
analyses. 

5.9.7  Resource  Requirements 

The  development,  programming  and 
testing  of  a  PMO  tailored  life  cycle  cost  model 
could  require  6  to  12  man-months  of  GS-12  to 
GS-13  level  analyst  effort.  However,  this  time 
can  be  significantly  reduced  if  an  existing  LCC 
model  can  be  found  and  tailored  to  the  PMO. 
Several  general  purpose  life  cycle  cost  models 
are  available  and  were  designed  to  be  tailored 
to  specific  PMO  needs.  The  Cost  Analysis 
Strategy  Assessment  (CASA)  models  were  de¬ 
veloped  for  and  distributed  by  the  Defense 
Systems  Management  College  (DSMC)  for 
this  purpose.  The  CASA  models  are  screen- 
oriented,  user-friendly  programs  and  can  be 
operated  on  microcomputers.  Use  of  these 


programs  could  be  quickly  mastered  by  GS-12 
level  analysts  using  the  users  guide  provided  by 
DSMC  with  copies  of  the  program.  The  most 
significant  task  associated  with  using  such 
models  is  obtaining  complete  and  valid  input 
data.  Input  data  requirements  may  include  key 
values  such  as  the  first  unit  production  costs. 
The  CASA  is  only  one  of  several  LCC  models 
available.  Program  management  office  person¬ 
nel  should  make  every  effort  to  find  the  LCC 
model  most  applicable  to  their  program  before 
initiating  efforts  to  modify  an  existing  LCC 
model  or  to  develop  a  new  model  from  scratch. 

5.9.8  Reliability 

Use  of  life  cycle  cost  models  for  analysis 
are  relatively  common  in  the  Department  of 
Defense  and  are  widely  accepted  as  a  quantita¬ 
tive  basis  for  decision-  making.  It  may  enhance 
the  credibility  of  a  PMO  analysis  in  the  view  of 
higher  levels  of  authority  if  an  LCC  model  is 
selected  that  has,  or  is  closely  related  to  one 
that  has  already  gained  acceptance.  Inquiries 
should  be  made  to  see  if  such  a  model  is  avail¬ 
able.  All  models  have  the  limitation  that  the  in¬ 
put  data  values  must  reflect  the  significant  and 
valid  differences  among  alternatives,  if  the 
model  is  to  produce  valid  and  useful  cost  dif¬ 
ferences  among  alternatives. 

5.10  COST  RISK/WBS 

SIMULATION  MODEL 

5.10,1  General 

This  technique  aggregates  cos*  risks  for 
each  of  several  WBS  elements  of  a  program 


into  the  total  program  cost  risk.  The  total  pro¬ 
gram  cost  risk  is  usually  expressed  as  a  cumula¬ 
tive  probability  distribution  of  total  program 
cost.  Such  distribution  information  can  be  used 
to  reflect  program  risk  by  computing  the  prob¬ 
ability  the  program  can  be  completed  for  a  spe¬ 
cific  dollar  value  or  less  and  what  level  of 
funding  would  be  required  to  have  a  given 
probability  of  completing  the  program  within 
the  available  funds.  A  micro  or  other  computer 
is  required  to  use  this  technique,  because  the 
analysis  requires  many  repeated  computations 
during  simulation  operations.  Similar  cost  risk 
analysis  can  be  performed  as  part  of  the  analy¬ 
sis  of  networks  by  such  models  as  VERT.  How¬ 
ever,  network  models  usually  require 
significantly  more  input  data  than  pure  cost 
risk/WBS  simulation  models. 

5.10.2  Description  of  Technique 

This  method  uses  the  Monte  Carlo 
simulation  analysis  method.  However,  vari¬ 
ations  of  the  technique  use  different  probabil¬ 
ity  distributions  to  describe  the  cost  risk 
associated  with  each  WBS  cost  element.  Uni¬ 
form,  Triangular,  Beta,  and  other  probability 
distributions  have  been  used  for  this  purpose. 
Use  of  the  Uniform  and  Triangular  distribu¬ 
tions  make  the  computation  easier.  However, 
use  of  the  Beta  distributions  allows  the  user 
more  freedom  in  describing  WBS  cost  element 
uncertainty.  Various  techniques  of  this  general 
type  differ  on  how  much  data  they  require  for 
each  WBS  cost  element  and  the  format  used 
to  present  analysis  results  and  assumptions 
with  respect  to  the  interdependence  among 
WBS  element  costs.  The  technique  uses  a  ran- 
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dom  number  generator  to  simulate  the  uncer¬ 
tainty  for  individual  WBS  elements.  Once 
costs  have  been  simulated  for  each  WBS  ele¬ 
ment,  they  are  aggregated  to  get  a  total  pro¬ 
gram  cost  estimate.  This  process  is  repeated 
many  times.  Each  time  a  new  set  of  WBS  ele¬ 
ment  costs  are  developed  is  called  an  “experi¬ 
ment”.  The  results  of  many  such  experiments 
provide  a  frequency  distribution  of  total  costs, 
reflecting  the  aggregate  of  the  cost  risks  associ¬ 
ated  with  all  the  individual  WBS  elements. 

5.10.3  When  Applicable 

Use  of  this  technique  is  applicable 
when  there  is  a  need  for  knowing  the  prob¬ 
ability  the  program  can  be  successfully  com¬ 
pleted  at  various  levels  of  funding.  It  is  also 
applicable  when  there  is  a  need  to  know  what 
level  of  funds  are  needed  to  achieve  a  specified 
probability  of  completing  the  program  at  that 
level  of  funding  or  less.  For  this  technique  to 
be  applicable,  it  is  also  necessary  to  be  able  to 
obtain  sound  estimates  of  the  cost  uncertainty 
associated  with  each  WBS  element  in  the  pro¬ 
gram.  When  a  cost  estimate  broken  out  by 
WBS  is  already  available,  it  is  a  relatively 
quick  analysis  procedure  to  use. 

5.10.4  Inputs  and  Outputs 

Inputs  and  outputs  vary  among  models 
implementing  this  type  of  analysis  technique. 
As  an  example  of  input  and  output  informa¬ 
tion,  a  simplified  version  of  the  Air  Force  Sys¬ 
tems  Command  (AFSC)  Risk  Model  will  be 
used.  The  AFSC  Risk  Model  is  probably  the 
most  widely  used  model  of  this  type  because  its 


use  has  been  directed  as  part  of  all  major  AFSC 
cost  estimates.  One  unique  aspect  of  the  AFSC 
Risk  Model  is  that  it  requires  four  estimates 
of  cost  uncertainty  for  each  WBS  element. 
However,  since  the  model  essentially  uses  only 
one  of  these  estimates  having  the  highest  risk, 
the  discussions  of  inputs  will  just  address  a  sin¬ 
gle  set  of  uncertainty  descriptive  data  for  each 
WBS  element. 

Inputs  -  For  each  model  run,  five  ele¬ 
ments  of  data  are  required  once  and  five  ele¬ 
ments  of  data  are  required  for  each  and  every 
WBS  cost  element  constituting  part  of  the  total 
program  cost  estimate.  They  are: 

•  For  each  model  run 

System  name 

Monte  Carlo  sample 
size  (default  value  is 
2500) 

Confidence  level 
computation  desired 
(default  value  is  90 
percent) 

Dollar  units  used 
for  inputs 

Date  of  run 

•  For  each  WBS  cost  element 


WBS  element  name 

Point  cost  estimate 
(most  likely) 

Low  end  of  cost 
range  value  (percen¬ 
tile  defined  by 
model) 

High  end  of  cost 
range  value  (percen¬ 
tile  defined  by 
model) 
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Level  of  WBS  ele¬ 
ment  cost  variance 
value  (judgment 
value  of  low,  medium 
low,  medium  high,  or 
high) 

Outputs  -  The  basic  WBS  simulation 
model  output  is  illustrated  by  Table  5.10-1.  It 
shows  into  which  of  60  sequentially  increasing 
cost  ranges  each  of  the  2500  simulated  total 
cost  estimates  fall.  As  an  example,  eight  of  the 
2500  simulation  experiments  produced  a  total 
cost  estimate  between  47.7  and  48.3  million 
dollars  and  thereby  fell  in  the  tenth  interval. 
Such  data  can  be  used  to  develop  total  cost 
probability  and  cumulative  probability  curves. 
Figure  5.10-1  is  an  example  of  such  a  cumula¬ 
tive  probability  curve  based  on  the  results  in 
Table  5.10-1.  The  same  data  can  also  be  used 
to  provide  output  information  with  respect  to 
the  confidence  level  that  a  program  can.be 
completed  for  a  specified  level  of  funding  or 
the  funding  required  to  achieve  a  specific  level 
of  confidence  that  the  program  will  cost  that 
value  or  less. 

5.10.5  Mqjor  Steps  In  Applying 
The  Technique 

The  first  step  in  applying  this  type  of 
technique  is  to  obtain  and  become  familiar 
with  one  of  several  available  computer  pro¬ 
grams  implementing  it  and  the  associated 
model  user  guidance.  It  will  seldom  be  practi¬ 
cal  or  desirable  to  develop  such  a  computer 
program  from  scratch. 


The  second  major  step  is  to  obtain  the 
input  data  required  by  the  specific  model  ob¬ 
tained  in  Step  1.  This  step  is  greatly  facilitated 
if  a  total  program  cost  estimate  is  already  avail¬ 
able,  broken  out  by  WBS  element.  If  such  an 
estimate  is  available,  the  required  WBS  cost 
element  uncertainty  input  data  can  generally 
be  obtained  by  interviewing  PMO  personnel. 
If  possible,  historical  cost  data  should  be  re¬ 
viewed  to  see  how  widely  similar  WBS  cost 
values  vary  on  other  programs.  The  third  step 
is  to  load  the  input  data  into  the  model  and 
make  one  or  more  model  runs  as  necessary. 
This  is  generally  far  less  time  consuming  than 
gathering  the  input  data. 

The  last  step  is  to  examine  the  model 
output  results  to  assure  they  appear  reasonable 

and  provide  the  type  of  information 
needed  to  show  how  WBS  element  risks  affect 
total  program  cost  risk. 

5.10.6  Use  of  Results 

The  primary  use  of  WBS  simulation 
model  results  is  to  show  how  WBS  element 
risks  may  cause  total  program  costs  to  vary 
from  the  point  estimate  used  for  budgets  and 
other  purposes.  It  can  also  be  used  to  compare 
estimated  costs  for  several  programs  at  a  speci¬ 
fied  confidence  level,  such  as  90  percent. 
Higher  headquarters  may  ask  to  see  such  in¬ 
formation  as  part  of  the  normal  review  process. 
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Table  5.10-1  Model  Output 


(EACH  INTERVAL  EQUALS  .63  MILLIONS) 


•INTERVAL 

RANGE 

FREQUENCY 

PROBABILITY 

1 

42.0000 

_ 

416333 

0 

.000 

2 

42.6333 

- 

43.2667 

0 

.000 

3 

43.2667 

- 

43.9000 

0 

.000 

4 

43.9000 

- 

44.5333 

0 

.000 

5 

44.5333 

- 

45.1667 

0 

.000 

6 

45.1667 

- 

45.8000 

0 

.000 

7 

45.8000 

- 

46.4333 

0 

.000 

8 

46.4333 

- 

47.0667 

0 

.000 

9 

47.0667 

- 

47.7000 

1 

.000 

10 

47.7000 

- 

48.3334 

8 

.003 

11 

48.3334 

- 

48.9667 

13 

.005 

12 

48.9667 

- 

49.6000 

9 

.004 

13 

49.6000 

- 

50.2334 

11 

.004 

14 

50.2334 

- 

50.8667 

19 

.008 

15 

50.8667 

- 

51.5000 

27 

.011 

16 

51.5000 

- 

52.1334 

45 

.018 

17 

52.1334 

- 

52.7667 

46 

.018 

18 

52.7667 

- 

53.4000 

48 

.019 

19 

53.4000 

- 

54.0334 

72 

.029 

20 

54.0334 

- 

54.6667 

57 

.023 

21 

54.6667 

- 

55.3000 

63 

.025 

22 

55.3000 

- 

55.9334 

89 

036 

23 

55.9334 

- 

56.5667 

101 

.040 

24 

56.5667 

- 

57.2000 

92 

.037 

25 

57.2000 

- 

57.8334 

112 

.045 

26 

57.8334 

- 

58.4667 

133 

.053 

27 

58.4667 

- 

59.1000 

130 

.052 

28 

59.1000 

- 

59.7334 

119 

.048 

29 

59.7334 

- 

60.3667 

135 

.054 

30 

60.3667 

- 

61.0001 

120 

.048 

31 

61.0001 

- 

61.6334 

134 

.054 

32 

61.6334 

- 

612667 

143 

.057 

33 

62.2667 

- 

619001 

99 

.040 

34 

62.9001 

- 

63.5334 

104 

.042 

35 

63.5334 

- 

64.1667 

106 

.042 

36 

64.1667 

- 

64.8001 

85 

.034 

37 

64.8001 

- 

65.4334 

60 

.024 

38 

65.4334 

- 

66.0667 

60 

.024 

39 

66.0667 

- 

66.7001 

41 

.016 

40 

66.7001 

- 

67.3334 

52 

.021 

41 

67.3334 

- 

67.9667 

50 

.020 

42 

67.9667 

- 

68.6000 

52 

.021 

43 

68.6000 

- 

69.2334 

22 

.009 

44 

69.2334 

- 

69.8667 

14 

.006 

45 

69.8667 

- 

70.5000 

2 

.001 

46 

70.5000 

- 

71.1334 

10 

.004 

47 

71.1334 

- 

71.7667 

7 

.003 

48 

71.7667 

- 

714000 

6 

.002 

49 

72.4000 

- 

73.0334 

1 

.000 

SO 

73.0334 

- 

73.6667 

1 

.000 

51 

73.6667 

- 

74.3000 

0 

.000 

52 

74.3000 

- 

74.9334 

0 

.000 

53 

74.9334 

- 

75.5667 

0 

.000 

54 

75.5667 

- 

76.2000 

1 

.000 

55 

76.2000 

- 

76.8334 

0 

.000 

56 

76.8334 

- 

77.4667 

0 

.000 

57 

77.4667 

- 

78.1000 

0 

.000 

58 

78.1000 

- 

78.7333 

0 

.000 

59 

78.7333 

- 

79.3667 

0 

.000 

60 

79.3667 

- 

80.0000 

0 

.000 

CUM  PROB 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

.000 

.003 

.008 

.012 

.016 

.024 

.035 

.053 

.071 

.091) 

.119 

.142 

.167 

.203 

.243 

.280 

325 

.378 

.430 

.478 

.532 

.580 

.634 

.691 

.731 

.773 

.815 

.849 

.873 

.897 

.913 

.934 

.954 

.975 

.984 

.990 

.991 

.995 

.998 

1.000 

1.000 

1.000 

1.000 

1.000 

1.000 

1.000 

1.000 

i. '  "i 

1.000 

1.000 
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UNCERTAINTY  DISTRIBUTION-CUMULATIVE  PROBABILITY 


Figure  5.10-1  Model  Output 


5.10.7  Resource  Requirements 

The  primary  resource  requirement  is  a 
copy  of  a  computer  program  implementing  this 
method  and  the  associated  user  guidance.  Air 
Force  experience  has  shown  that  GS-9  and 
above  cost  analysts  can  quickly  learn  to  run 
such  a  model,  if  supported  by  PMO  specialists 
*n  providing  WBS  cost  element  uncertainty 
range  and  level  judgments.  A  microcomputer 
is  also  required.  The  AFSC  risk  model  runs  on 
a  Zenith  100  computer.  Other  similar  models 
run  on  IBM  PCs. 


5.10.8  Reliability 

The  mathematics  and  logic  of  the  WBS 
simulation/cost  risk  technique  is  generally 
sound.  An  exception  is  that  these  models  gen¬ 
erally  do  not  fully  address  the  interactions  be¬ 
tween  WBS  elements.  They  usually  assume 
either  total  dependence  or  total  independence 
among  WBS  elements,  The  true  situation  will 
probably  vary  from  program  to  program  and 
will  almost  always  be  somewhere  between  total 
independence  and  total  interdependence.  The 
greatest  limitation  of  this  method  is  the  diffi- 
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culty  in  obtaining  sound  and  supportable  in¬ 
put  values. 

5.11  RISK  FACTORS 

5.11.1  General 

This  method  is  often  quite  simple  to 
apply  except  for  the  difficulty  in  obtaining 
sound  and  dependable  input  values  to  describe 
the  risk  associated  with  each  WBS  element. 
Often  the  input  values  are  quick  judgments 
made  by  PMO  personnel.  The  method  does  not 
include  procedures  for  systematic  and  scientific 
development  of  the  needed  input  data.  The 
primary  use  of  the  method  is  to  estimate  the 


total  added  program  costs  that  might  be  ex¬ 
pected  due  to  risks  associated  with  the  various 
program  WBS  elements. 

5.11.2  Description  of  Technique 

The  basic  concept  of  the  Risk  Factor 
method  is  to  determine  factors,  or  multipliers, 
with  which  to  increase  individual  baseline  WBS 
element  cost  estimate  to  cover  additional  costs 
resulting  from  risks.  The  objective  of  using  this 
method  is  to  determine  a  reasonable  budget, 
above  that  resulting  from  a  baseline  cost  esti¬ 
mate,  to  cover  anticipated  risk  associated  cost 
growth.  The  method  uses  a  WBS  or  cost  break¬ 
down  structure  based  on  a  technical  break-  • 
down  like  that  shown  in  Figure  5.11-1. 


Figure  5.11-1  Cost  Breakdown  Structure 


The  baseline  cost  estimate  must  have 
been  developed  for  each  cost  element.  Apply¬ 
ing  whatever  considerations  are  appropriate,  a 
risk  factor  is  established  for  each  cost  element. 
This  factor  will  generally  be  a  value  between 
1.0  and  2.0  with  1.0  indicating  no  risk  and  2.0 
indicating  so  much  risk  that  expected  costs 
would  be  twice  the  baseline  cost  estimate  val¬ 
ues.  Every  baseline  WBS  cost  estimate  is  then 
multiplied  by  its  risk  factor  to  obtain  the  new 
WBS  element  cost  estimates.  These  new  esti¬ 
mates  summed  to  get  a  budget  value,  which 
provides  a  level  of  funding  which  will  account 
for  technical  or  other  risk. 

The  obtaining  of  sound  WBS  element 
risk  factors  is  the  key  feature  of  this  method 
and  may  be  difficult.  There  is  little  docu¬ 
mented  experience  upon  which  analysts  can 
draw  in  order  to  substantiate  such  factors. 
Since  these  factors  have  a  significant  impact 
on  the  analysis  results,  it  is  important  that  the 
inputs  be  obtained  from  highly  experienced 
technical  experts.  In  other  words,  the  apparent 
simplicity  of  the  method  has  not  relaxed  the  re¬ 
quirement  that  the  most  experienced  PMO 
personnel  take  key  roles  in  the  analysis.  Once 
a  baseline  cost  estimate  has  been  prepared  us¬ 
ing  cost  estimating  methods,  an  analyst  should 
be  able  to  prepare  a  new  cost  estimate  using 
risk  factors  in  a  relatively  short  time.  The 
length  of  time  will  depend  on  the  difficulty  an 
analyst  has  in  obtaining  the  assistance  of  tech¬ 
nical  experts,  and  on  how  detailed  a  WBS  or 
cost  breakdown  is  involved. 


5.113  When  Applicable 

The  survey  of  PMOs  on  past  and  current 
risk  analysis  utilization  showed  that  only  six 
out  of  the  fifty-seven  PMOs  responding  had 
used  this  technique.  These  six  PMOs  found  the 
technique  useful  primarily  for  POM/BES 
preparation  and  program  planning.  The  tech¬ 
nique  is  more  applicable  early  in  the  life  of  a 
program  when  information  is  not  available  to 
apply  some  of  the  more  sophisticated  risk 
analysis  techniques.  This  technique  is  only  ap¬ 
plicable  when  a  point  cost  estimate,  broken  out 
by  WBS  element,  is  available.  The  method’s 
simplicity  makes  it  applicable  to  even  small, 
low  cost  programs. 

5.11.4  Inputs  and  Outputs 

Inputs  -  One  primary,  and  generally 
available,  input  of  a  risk  factor  assessment  is  a 
baseline  cost  estimate  broken  out  by  WBi  ele¬ 
ment.  The  second  primary  input  is  a  set  of  risk 
factors  for  each  WBS  cost  element.  These  fac¬ 
tors  will  usually  be  subjective  judgments  of  ex¬ 
perienced  personnel  who  know  the  program, 
its  current  status,  and  potential  problem  areas. 
The  use  of  check  or  watch  lists  and  the  num¬ 
ber  of  items  in  the  list  that  apply  to  each  WBS 
element  is  one  way  of  helping  make  a  judg¬ 
ment  of  the  level  of  risk  associated  with  each 
element. 

Output  -  The  output  of  a  risk  factor  ap¬ 
plication  is  a  budget  or  cost  estimate  increased 
over  the  baseline  budget  (or  estimate)  by  an 
amount  required  to  cover  risk  induced  costs. 
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5.11.6  Use  of  kesults 


5.11.5  M^jor  Steps  in  Applying 
the  Technique 

The  major  steps  in  applying  the  tech¬ 
nique  are: 

•  Obtain  a  program  cost  esti¬ 
mate  broken  out  by  WBS 
element.  Such  estimates 
should  be  available  and 
their  preparation  is  not  con¬ 
sidered  to  be  part  of  apply¬ 
ing  this  method. 

•  For  each  WBS  element 
obtain  an  estimate  for  the 
percent  of  additional  costs 
that  should  be  added  to 
accommodate  additional 
work  resulting  from  risks. 

The  opinions  of  knowl¬ 
edgeable  technical  and  ex¬ 
perienced  program 
management  should  be 
sought  and  used.  Review¬ 
ing  the  lessons  learned  for 
similar  systems  could  also 
provide  insight  into  how 
much  risk  might  be  in¬ 
volved.  If  similar  things 
have  been  done  before, 
and  by  the  same  people 
assigned  to  the  current 
irogram,  risks  should  be 
ower.  It  must  be  remem¬ 
bered  that  past  programs 
were  also  risky  and  there¬ 
fore  parametric  cost  esti¬ 
mates  based  thereon  also 
include  some  costs  to 
cover  risk. 

•  Recalculate  the  total  pro¬ 
gram  costs  by  summing  all 
the  WBS  element  costs, 
each  of  which  has  been 
adjusted  by  the  associated 
factor  percentage  increase 
to  accommodate  the  risks 
associated  with  it. 


According  to  the  survey  of  PMO  risk 
analysis  applications  one  or  more  PMOs  found 
the  results  of  risk  factors  analysis  of  some  sig¬ 
nificant  use  for  POM/BES  preparation,  pro¬ 
gram  status  reporting,  program  planning  and 
DAB  milestone  briefings.  This  method  has  also 
been  used  to  support  U.S.  Army  TRACE  cost 
risk  procedures. 

5.11.7  Resource  Requirements 

Resource  requirements  for  this  method 
can  be  quite  variable.  Frequently,  the  same 
cost  estimator  responsible  for  preparing  the 
baseline  cost  estimate  can  also  provide  the  ad¬ 
ditional  risk  factor  results  in  a  few  hours  if  he/ 
she  is  provided  the  WBS  element  factors  by 
appropriate  experts  in  a  timely  manner.  How¬ 
ever,  application  of  the  method  can  become 
more  involved  as  more  technical  and  other  ex¬ 
perts  are  used  to  derive  the  individual  WBS  ele¬ 
ment  risk  factors. 

5.11.8  Reliability 

The  reliability  of  this  technique  can  vary 
widely  both  in  fact  and  in  the  judgment  of 
those  reviewing  the  results.  Since  use  of  the 
technique  generally  requires  judgments  based 
on  limited  information,  the  knowledge  and 
skill  of  those  making  the  judgments  will  greatly 
affect  the  reliability  of  the  results.  A  quick 
analysis,  where  the  risk  level  factor  judgments 
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for  all  WBS  elements  are  made  by  a  single  cost 
analyst,  without  inputs  from  technical  and 
other  experts,  would  very  likely  produce  rela¬ 
tively  low  reliability  results.  The  reliability  of 
this  method  is  increased  by  providing  docu¬ 
mented  justification  for  all  WBS  element  fac¬ 
tor  values  used. 

5.12  PERFORMANCE  TRACKING 

5.12.1  General 

Much  has  been  written  about  technical 
risk.  The  GAO  report  on  technical  risk,  April 
1986,  spent  a  great  deal  of  time  discussing  the 
importance  of  managing  the  technical  aspects 
of  a  program.  However,  measuring  technical 
risk  on  any  effort  that  involves  furthering  the 
state-of-the-art  is  a  very  difficult  task,  which 
in  and  of  itself,  can  involve  a  great  amount  of 
risk.  There  are  some  concrete  measurements 
that  can  be  useful  in  measuring  technical  ad¬ 
vancement  progress  against  preset  goals  of 
programs.  Many  of  these  are  described  in  a 
publication  entitled  "Technical  Risk  Assess¬ 
ment:  Staying  Ahead  of  Real-Time  Technical 
Problems,  Visibility  and  Forecasts”  (currently 
in  draft  form).  This  is  a  Navy  document  re¬ 
leased  in  March  1986  (Ref.  5-2).  Within  the 
document  are  several  recommended  measures 
for  evaluating  technical  progress. 

5.12.2  Description  of  Technique 

The  technique  advocates  the  use  of  a 
Technical  Risk  Assessment  Report,  which  is 


updated  monthly.  The  report  is  based  on  work¬ 
ing  level  data  but  is  intended  to  provide  an 
overview  of  current  trends  and  status.  The  tech¬ 
nique  uses  a  set  of  standard  technical  indicators 
which  have  been  proven  to  be  effective  meas¬ 
ures  of  technical  performance.  In  addition  to 
the  standard  measures,  program  unique  tech¬ 
nical  indicators  are  also  developed.  Each  of  the 
measures  has  clearly  defined  performance  pro¬ 
jections  and  pre-set  alert  criteria.  The  standard 
indicators  are  shown  in  Figure  5.12-1  and  a 
sample  indicator  is  shown  in  Figure  5.12-2. 

5.12.3  When  Applicable 

The  performance  tracking  technique  is 
most  useful  when  there  are  specific  criteria  es¬ 
tablished  that  are  objective  and  quantifiable.  It 
can  best  be  utilized  for  the  management  of 
near  term  requirements.  The  approach  can  be 
used  with  minor  modifications  on  any  type  of 
program  and  could  be  used  in  conjunction  with 
more  elaborate  probabilistic  risk  models  that 
can  examine  the  corresponding  cost  and  sched¬ 
ule  impacts  of  current  technical  performance. 

5.12.4  Inputs  and  Outputs 

The  technique  requires  that  perform¬ 
ance  be  tracked  on  a  monthly  basis  for  each 
technical  indicator  selected.  This  requires  full 
cooperation  with  the  contractor  and  his  active 
participation  in  managing  risk  (a  good  benefit). 
The  output  can  be  in  the  form  of  a  risk  manage¬ 
ment  report  or  a  briefing.  The  contents  should 
contain  an  analysis  of  each  of  the  indicators 
current  performance  and  longer  term  trend. 
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Figure  5.12-1  Standard  Indicators 
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WEIGHT 

PURPOSE:  To  show  worst  case  and  most  likely  weight  estimates 
compared  to  specific  goal  and  reduction  plan 

DATA  GROUND  RULES: 

-  Most  likely  estimate  for  system  based  on  sum  of 
most  likely  estimates  for  subsystems 

-  Worst  case  estimate  for  system  based  on  sum  of 
worst-case  estimates  for  subsystems 

ALERT  ZONES: 

GREEN  -  Both  estimates  less  than  reduction  plan 
YELLOW  -  Worst  case  estimate  greater  than  reduction  plan 
RED  -  Most  likely  estimate  greater  than  reduction  plan 


Figure  5.12-2  Sample  Indicators 
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5.12.5  Mqjor  Steps  in  Applying 
the  Technique 

One  of  the  first  steps  in  adapting  the 
technical  risk  assessment  method  to  track  risk 
performance  is  to  select  the  standard  indicators 
that  can  be  applied  to  the  development  pro¬ 
gram.  Many  of  the  standard  indicators  (Figure 

5.12- 1)  can  be  used  on  development  pro¬ 
grams,  and  the  utility  of  certain  indicators  will 
vary  as  the  program  progresses.  In  the  case  of 
an  airborne  system  weight  and  size  are  always 
significant.  Weight  and  size  may  not  be  as  sig¬ 
nificant  on  a  system  to  be  installed  aboard  air¬ 
craft  carriers,  however,  if  the  system  is 
submarine  installed,  size  again  becomes  im¬ 
portant. 

The  selection  of  indicators  should  in¬ 
clude  ones  for  the  entire  program  and  selected 
ones  for  the  subsystems.  The  unusual  aspects 
of  a  developmental  program  frequently  re¬ 
quire  the  use  of  special  technical  indicators. 

In  the  case  of  space  systems,  certain  in¬ 
dicators  are  appropriate  such  as  the  production 
of  gasses  from  the  material  in  the  product 
when  exposed  to  a  space  environment.  Figure 

5.12- 3  shows  some  potential  special  indica¬ 
tors. 

Each  indicator,  whether  standard  or 
special  must  have  ground  rules  established  for 
data  collection  and  assessment.  This  can  be  in 
the  form  of  a  dictionary  and  describe  the  ob¬ 
ject  of  the  indicator,  why  it  was  chosen,  the  use 
of  the  indicator  and  what  is  to  be  done  when  a 
signal  is  generated  that  indicates  a  problem  is 


developing.  It  should  be  in  sufficient  detail  to 
inform  the  system  operator  of  the  meaning  of 
the  indicator  and  the  relationship  of  the  meas¬ 
urement  to  risk. 

It  is  advisable  to  predict  the  trends  that 
might  be  expected  during  the  life  of  the  indica¬ 
tor.  Expected  values  may  take  many  different 
forms  or  curve  functions  but  should  have  a 
traceability  to  the  program  goals,  either  cost, 
schedule,  performance,  or  various  combina¬ 
tions.  Evaluation  criteria  must  be  set  so  as  to 
flag  a  situation  that  can  signal  a  problem.  Color 
coding  such  as  red,  yellow  or  green  for  high, 
medium,  or  low  risk  can  be  used  as  well  as  per¬ 
centage  bands  for  the  same  type  of  message. 
These  bands  may  vary  as  time  progresses,  that 
is,  getting  tighter  as  completion  is  nearing  or 
getting  more  tolerant  as  time  passes  to  indicate 
a  risk  that  is  disappearing.  In  any  case,  both  the 
program  office,  contractor,  and  subsystem  con¬ 
tractors)  should  agree  and  understand  the  tol¬ 
erance  bands  and  their  significance  in  order  to 
facilitate  rapid  corrective  action. 

All  the  above  would  be  useless  unless  a 
formal,  contractually  required  reporting  sys¬ 
tem  is  used.  This  could  be  in  different  form,  ac¬ 
cording  to  the  type  of  the  program  and  the  style 
of  the  manager.  It  may  be  produced  in  vu- 
graphs  in  a  manner  immediately  usable  by  the 
government  manager  for  required  higher  level 
periodic  briefings  or  in  a  raw  form  as  numerical 
data  points.  In  any  case,  it  must  be  in  a  form  im¬ 
mediately  applicable  by  both  the  contractor 
and  the  program  manager  in  making  decisions 
affecting  the  program. 
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INDICATORS  DERIVED  FROM 
SPECIFICATION  REQUIREMENTS 

INDICATORS  DERIVED  FROM 
PROGRAM  REQUIREMENTS 

PERFORMANCE  CHARACTERISTICS 

SCHEDULE 

•  SPEED,  RANGE,  CAPACITY, 
ACCURACY,  ETC. 

•  FEASIBILITY/PROBABILITY 

OF  TIMELY  ACCOMPLISHMENT,  ETC. 

PHYSICAL  CHARACTERISTICS 

RESOURCES 

•  CENTER  OF  BUOYANCY, 

LENGTH,  ETC. 

•  ADEQUACY,  DISTRIBUTION.  ETC. 

EFFECTIVENESS  CHARACTERISTICS 

TEST  PLAN 

•  RELIABILITY,  SAFETY, 

LOGISTICS  SUPPORT,  ETC. 

•  SUFFICIENCY  OF  PLANNED 

TESTING,  ETC. 

ENVIRONMENTAL  CONDITIONS 

PROCUREMENT  FACTORS 

•  VIBRATION,  TEMPERATURE, 

SHOCK,  ETC. 

•  AVAILABILITY  OF  MULTIPLE 

SOURCES,  ETC. 

DESIGN  AND  CONSTRUCTION 

•  TECHNOLOGY,  PACKAGING, 
MATERIALS,  ETC. 

Figure  5.12-3  Sample  Special  Indicators 


As  in  any  system  that  requires  the  coor¬ 
dinated  efforts  of  contractors  and  government 
technical  and  management  personnel,  it  is  nec¬ 
essary  to  place  someone  in  charge  of  insuring 
that  the  job  is  being  done  accurately  and  in  a 
timely  fashion  and  that  the  proper  decision¬ 
makers  are  informed  of  the  risk  situations. 

In  summary,  the  major  steps  in  applying 
risk  measurement  techniques  are: 

1)  Applying  the  standard  indi¬ 
cators 

2)  Selecting  special  indica¬ 
tors 

3)  Establishing  data  defini¬ 
tions 

4)  Projecting  expected  trends 

5)  Setting  the  evaluation  cri¬ 
teria 


6)  Planning  the  reporting 
system 

7)  Assigning  responsibilities. 

5.12.6  Use  of  Results 

The  technical  risk  assessment  reports 
furnish  the  information  needed  to  start  any 
action  that  might  be  required  to  correct  poten¬ 
tial  problems.  Each  indicator  should  be  exam¬ 
ined  separately  and  then  examined  as  related 
groups  of  indicators.  In  using  the  results,  the 
factors  of  cost,  schedule,  and  technical  risks 
must  be  examined  simultaneously. 

5.12.7  Resource  Requirements 

This  technique  requires  people  with 
sufficient  knowledge  and  skills  in  highly  spe¬ 
cialized  technical  areas.  The  data  received 


emanates  from  many  functional  groups  includ¬ 
ing  fabrication,  assembly,  engineering,  quality 
control,  etc.  and  must  be  analyzed  by  people 
who  have  these  skills  and  can  make  technical 
analytical  assessments  of  the  reports.  This  does 
not  mean  that  each  functional  risk  assessment 
area  requires  a  full  time  person.  While  system 
start-up  costs  vary,  it  should  not  require  more 
than  1-2  man-months  of  effort.  Typically,  the 
sustaining  costs  are  estimated  tc  be  a  one  per¬ 
son  effort  for  a  fairly  large  program. 

5.12.8  Reliability 

In  order  to  have  a  reliable  technical  risk 
assessment,  it  is  necessary  that  all  major  par¬ 
ticipants  understand  the  importance  of  the  as¬ 
sessment  and  be  actively  involved  in 
establishing  and  implementing  the  system. 
Each  member  of  the  team  should  be  involved 
in  the  initial  assessment  of  the  programs  tech¬ 
nical  risk  and  help  in  the  selection  of  the  indica¬ 
tors  used  in  tracking  the  risk.  These  are  the 
same  people  that  should  be  providing  the  up¬ 
dates  for  each  reporting  period.  The  early  sur¬ 
facing  of  potential  problems  anticipates  the 
problem  prior  to  failure  and  with  proper  man¬ 
agement  action,  failure  may  be  precluded  or  at 
least  tempered. 

5.12.9  Performance  Tracking  - 
Supplemental  Information 

Performance  tracking  is  not  new.  It  has 
existed  in  one  form  or  another  for  many  years, 
but  recently  it  has  gained  in  popularity  and  use. 
There  are  many  variations  on  the  theme  pre¬ 
sented  in  the  above  discussion.  Since  control  is 


one  of  the  most  critical  elements  in  risk  man¬ 
agement,  and  performance  tracking  is  one  of 
the  most  effective  control  techniques,  another 
variation  of  the  method  is  presented  below. 

Fully  integrated,  performance  measure¬ 
ment  -  is  a  capability  being  developed  to  inte¬ 
grate  technical  performance,  schedule 
performance,  and  cost  performance.  It  is  also 
aimed  at  providing  Earned  Value  performance 
measurement  capability  to  Government  pro¬ 
gram  offices  that  are  not  getting  formal  con¬ 
tractor  performance  data.  The  major  steps  are 
as  follows: 


-  Technical  Performance  - 

•  From  program  direction, 
plans,  ana  specifications, 
identify  specific  technical  pa¬ 
rameters  and  their  value  for 
performance,  producibilify, 
quality  assurance,  reliability, 
maintainability,  support,  etc. 
A  few  examples  are  shown 
in  Figure  5.12-4 

•  Relate  each  of  these  techni¬ 
cal  parameters  to  specific 
WBS  elements  whenever 
practical.  Many  of  them  will 
only  relate  to  the  total  sys¬ 
tem  level,  but  many  will 
come  from  the  Spec  Tree 
which  should  match  the 
Work  Breakdown  Structure. 

•  Define  specific  methods 
for  calculating,  measuring, 
or  observing  the  value  of 
each  technical  parameter. 

•  Assign  a  specific  individ¬ 
ual  or  organization  the  re¬ 
sponsibility  for  managing 
the  technical  parameter 
and  the  progress  toward 
achieving  the  goal  value. 


PERFORMANCE 

Speed  (KTS) 

Weight  (Lbs) 

Range  (NM) 

Power  (KW) 

Turn  Rate  (Deg/Sec) 
Takeoff  Distance  (Ft) 

Climb  Rate  (Ft/Sec) 
Accuracy/CEP  (Ft) 

Radar  Cross  Section  (Sq  Ft) 


RELIABILITY 

MTBF  (Hrs/Days) 

MTTR  (Hrs/Days) 

LRU  vs  SRU  (%) 
Probability  of  Component/ 
Assy.  Failure  (0  -  1.0) 
Life  Cycle  Analysis  ($) 
Design  to  Cost  ($) 


PRODUCIBILITY 
Capital  ($) 

Manpower  (People  Count) 
Facilities  (Sq  Ft) 

Material  ($) 

Equipment  (Machinery  Req’d) 
Schedule  (Time) 

Risk  (0-1.0) 


MAINTAINABILITY 

Standardization  (%) 
Modularity  (%) 

Update  Ability  (0  -  1,0) 
Special  Equipment  ($) 

STE  (S) 

Frequency  (Schedule)  (Time) 
Costs  (S) 


QUALITY  ASSURANCE 

Scrap,  Rework  &  Repair 
(%  of  Labor) 

Yield  (%  of  1st  Time 
Inspection  Successes) 
Supplier  Rating  (%) 

Quality  Costs  (S) 

Customer  Satisfaction  (0  -  1.0) 
Software  (LOC  in  Violation 
per  1000  LOC) 


SUPPORTABIUTY 

Parts  Inventory  ($) 

Costs  ($; 

Resources  (Manpower, 
Equipment,  Facilities) 
Modularity  (%) 

Operational  Availability  (%) 
MTBF  (Hrs/Days) 

MTTR  (Hrs/Days) 


Figure  5.12-4  Fully  Integrated  Performance  Measurement 
Typical  Technical  Parameters 


-  Schedule  Performance  - 

•  Identify  (or  create)  specific 
schedule  events  at  which 
calculation  or  observation 
is  to  be  made. 

•  Determine  values  or  con¬ 
ditions  that  should  be 
achieved  by  each  mile¬ 
stone.  Also  set  a  tolerance 
or  “alarm”  value  to  repre¬ 
sent  a  threshold  for  cor¬ 
rective  action. 

•  Identify  (or  create)  a  spe¬ 
cific  schedule  event  at 
which  the  goal  is  to  be 
achieved. 

•  A  plot  of  the  technical 
performance  parameter 
value  against  time  gives  a 
visual  portrayal  of  the  re¬ 
lationship  between  techni¬ 
cal  performance  and 


schedule.  See  Figures 
5.12-5  and  5.12-6. 


-  Cost  Performance  - 

•  Assign  budgets  to  each 
technical  performance  pa¬ 
rameter.  These  budgets 
may  be  real  and  add  up  to 
contractual  values  or  ficti¬ 
tious  units  just  to  determine 
relative  weights.  There  are 
many  different  ways  to  as¬ 
sign  these  budgets.  The 
only  requirement  is  ration¬ 
ality,  traceability,  and  con¬ 
sistency. 

•  Distribute  the  assigned 
budgets  to  each  of  I the 
measurement  milestones 
based  on  engineering 
judgment  of  the  percent 
of  the  total  value  associ¬ 
ated  with  each  milestone. 
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Figure  5.12-5  Technical  Performance  Schedule  Milestones 
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Figure  5.12-6  Technical  Performance  Management 
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•  Use  conventional  earned 
value  techniques  to  meas¬ 
ure  accomplishment  (e.g. 

50/50  milestones). 

•  Apply  the  schedule  per¬ 
formance  index  to  appro¬ 
priate  activities  in  the 
resource  loaded  network 
to  determine  the  cost  im¬ 
pact  of  the  technical  and 
schedule  performance. 

A  quick  example  may  help  clarify  the 
technique.  Referring  to  Figure  5.12-5,  per¬ 
formance  parameter  #1  has  a  numeric  goal.  A 
method  of  calculating  progress  against  the  goal 
has  been  derived.  At  selected  milestone  1,  pro¬ 
gress  against  th?  goal  is  calculated.  By  selected 
milestone  3,  progress  against  the  goal  can  actu¬ 
ally  be  observed,  and  by  milestone  5,  the  goal 
should  be  attained. 


5.13  OTHER  COMMON  TECHNIQUES 

5.13.1  CPR  Analysis 

Cost  Performance  Reports  (CPR)  ob¬ 
tained  to  comply  with  DoDI  7000.10  have  be¬ 
come  useful  in  uncovering  areas  where 
technical  problems  are  causing  variances.  In 
this  report  the  contractor  explains  cost  and 
schedule  variances  by  means  of  a  narrative  de¬ 
tailing  the  specific  problem  that  has  caused  the 
variance.  Many  of  the  variances  reported  can 
signal  risk  situations  as  they  are  developing, 
such  as  late  vendor  or  sub-contractor  deliver¬ 
ies.  The  continuation  of  these  types  of  sched¬ 
ule  slips  can  put  an  entire  program  schedule  at 
risk.  Normally,  Government  program  manag¬ 
ers  are  limited  in  what  they  can  do  to  alleviate 


these  situations  except  in  cases  where  Govern¬ 
ment  Furnished  Properties  (GFP)  are  causing 
the  delays.  The  GFP  shortage  situation  can 
sometimes  be  alleviated  by  high  level  coordi¬ 
nation  with  the  supplying  Government  agency. 
However,  this  does  not  always  work.  For  exam¬ 
ple,  DoD  control  of  DOE  supplied  special 
weapons  is  not  very  effective  and  the  risk  of 
late  warhead  specifications  in  terms  of  weight 
and  size  can  cause  significant  risk  in  the  sched¬ 
ule  of  the  carrying  vehicle.  Schedule  problems 
of  this  type  invariably  cause  cost  problems  as 
vehicle  designers  scramble  to  modify  designs  to 
accommodate  changing  specifications.  The 
risk  in  this  situation  is  that  rapid  advancement 
of  the  design  cycle  to  meet  original  target  dates 
can  be  affected  by  late  breaking  specification 
changes. 

Cost  variances  can  also  be  risk  involved, 
as  large  cost  growth  can  jeopardize  a  program 
to  the  point  of  causing  cancellations.  It  is  naive 
to  not  consider  cost  growth  as  a  significant  risk 
item.  The  CPR  is  designed  to  display  cost 
growth  as  a  variance  and  then  discuss  the  vari¬ 
ance  in  terms  of  cause,  impact,  and  any  correc¬ 
tive  action  that  might  be  taken  to  alleviate  the 
situation. 

If  the  program  is  receiving  the  CPR  it 
should  be  used  for  risk  assessment  and  analy¬ 
sis  by  the  program  manager.  The  discussion  of 
variances  in  the  format  five  report  can  contain 
data  that  permits  the  determination  of  items 
that  may  be  presenting  new  and  previously  un¬ 
discovered  risks.  These  risks  should  then  be  in¬ 
vestigated  to  ascertain  their  effects  on  the 
program. 
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5.13.2  Independent  Technical 
Assessment 

General  -  An  independent  technical  as¬ 
sessment  requires  people  other  than  those  un¬ 
der  control  of  the  PMO  and  therefore  will 
always  require  approval  by  some  higher  level 
of  authority.  The  timing  of  such  reviews  is  criti¬ 
cal.  If  problems  are  found,  there  must  be  time 
to  correct  them  before  any  critical  milestone 
reviews.  This  technique  has  been  used  by  a 
multi-service  program  and  is  cited  as  substan¬ 
tially  reducing  program  risk,  especially  that 
which  was  associated  with  tri-service  involve¬ 
ment. 

Description  of  Technique  -  This  tech¬ 
nique  involves  a  team  of  experts  from  outside 
the  PMO  reviewing  a  number  of  specified  as¬ 
pects  of  the  program.  The  team  usually  con¬ 
sists  of  very  senior  personnel  who  can  make 
timely  evaluations  of  PMO  activities  and  pro¬ 
gress  based  on  their  extensive  experience. 
Such  a  team  can  vary  in  size  depending  on  the 
size  of  the  program  and  how  many  issues  the 
team  has  been  chartered  to  review.  The  entire 
process  is  usually  limited  to  four  to  eight  weeks 
of  near  full  time  effort.  The  final  product  is  al¬ 
most  always  a  briefing  to  the  level  of  authority 
authorizing  the  review  and  sometimes  a  written 
final  report. 

When  Applicable  -  An  acceptable  time 
to  use  the  technique  is  in  support  of  design  re¬ 
views.  It  can  also  be  used  to  quiet/end  percep¬ 
tions  of  a  troubled  program.  A  good  time  for  an 
independent  technical  assessment  is  when  a 
program  is,  or  is  perceived  to  be  in  trouble  and 


critics  have  become  vocal.  If  the  trouble  is  real, 
this  technique  will  give  the  PMO  added  credi¬ 
bility  and  quiet  critics.  When  possible,  such  re¬ 
views  should  be  scheduled  to  cause  minimum 
disruption  of  milestone  activities.  An  inde¬ 
pendent  technical  assessment  is  usually  more 
appropriate  during  system  development  than 
during  production. 

Inputs  and  Outputs  -  The  inputs  will 
vary  widely  depending  on  the  issues  shown  to 
be  addressed  and  the  expertise  of  the  team 
members.  Team  members  will  obtain  the  infor¬ 
mation  they  need  through  briefings  by  PMO 
personnel,  review  of  PMO  documents,  inter¬ 
views  and  visits  to  contractors’  facilities.  The 
expertise  and  experience  team  members  bring 
with  them  to  the  team  is  an  important  input. 
The  most  common  output  is  a  briefing  to  the 
commander  authorizing  the  review  and  to  oth¬ 
ers  as  appropriate.  The  briefing  must  address 
each  of  several  criteria  or  issues  defined  at  the 
onset  of  the  review.  It  should  also  include  rec¬ 
ommendations  for  follow-on  action. 

Major  Steps  in  Applying  the  Technique  - 
The  following  steps  are  common  to  most  in¬ 
dependent  technical  assessments: 

•  Direction  by  a  higher  level 
of  authority  with  control  of 
or  access  to  the  required  ex¬ 
pert  resources,  to  conduct 
the  review 

•  Specification  of  the  issues 
to  be  addressed 

•  Formation  of  the  review 
team 

•  Gathering  the  required  in¬ 
formation  about  PMO  ob- 
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jectives,  problems,  status, 
resources,  and  activities 

•  Analyzing  the  information 
gathered 

•  Presenting  the  results  to 
the  authority  who  re¬ 
quested  the  review  and  to 
others  as  appropriate. 

Uses  of  Results  -  Independent  technical 
assessments  are  useful  for  design,  acquisition 
strategy,  planning,  and  implementation  coordi¬ 
nation.  When  the  review  results  are  favorable, 
there  is  instant  program  risk  reduction  with  as¬ 
sociated  benefits  in  meeting  pending  milestone 
reviews. 

Resources  Required  -  Resources  of  two 
types  are  required  to  carry  out  an  independent 
technical  assessment.  First,  a  team  of  up  to  10 
experts  is  needed  to  form  the  review  team.  The 
people  required  must  be  experienced  and  cer¬ 
tainly  would  include  some  or  all  at  the  GS-15 
level  or  above.  These  people  would  probably 
have  to  commit  two  to  four  weeks  of  effort  to 
the  team  over  a  period  of  four  to  eight  weeks. 

In  addition  to  team  resource  require¬ 
ments,  the  PMO  has  to  provide  a  number  of  in¬ 
formational  briefings  and  interviews  to  quickly 
provide  the  review  team  with  the  required  in¬ 
formation.  Where  members  of  the  review  team 
are  visiting  from  out  of  town,  the  PMO  may  be 
required  to  perform  substantial  protocol  and 
administrative  tasks.  The  PMO  usually  pays  all 
travel  costs  for  team  members. 

Reliability  -  The  reliability  of  an  inde¬ 
pendent  technical  assessment  is  usually  high. 
The  reliability  somewhat  depends  on  the 


quality  of  the  team  members,  that  is  their  rec¬ 
ognized  level  of  expertise.  While  team  inde¬ 
pendence  is  essential,  cooperation  and  trust 
between  the  team  and  the  PMO  is  also  essen¬ 
tial.  The  PMO  must  provide  all  required  infor¬ 
mation  and  the  review  team  must  present  a 
balanced  picture  while  not  just  focusing  on  the 
most  negative  areas.  The  major  disadvantage 
of  an  independent  technical  assessment  is  that 
for  a  time  it  can  disrupt  PMO  activities.  This  is 
especially  true  if  it  points  out  deficiencies  that 
must  be  fixed,  and  there  is  no  time  to  make 
the  needed  fixes  prior  to  an  important  mile¬ 
stone.  Therefore,  the  timing  of  the  review  is 
important  and  should  be  considered  in  the 
planning  for  such  reviews. 

5.133  Independent  Cost  Estimates 

Independent  cost  estimates  must  be  ac¬ 
complished  one  or  more  times  for  many  DoD 
programs  in  accordance  with  the  requirements 
of  the  DoD  Independent  Cost  Analysis  (ICA) 
program. 

The  ICA  program  came  about  because 
of  a  perception  within  the  Office  of  the  Secre¬ 
tary  of  Defense  (OSD),  that  PMOs,  because  of 
their  commitment  to  achieving  program  goals, 
naturally  tend  to  be  optimistic  regarding  the 
risks  and  costs  of  program,  particularly  in  early 
stages.  To  provide  OSD  and  senior  service  de¬ 
cision-makers  with  data  reflecting  an  inde¬ 
pendent  viewpoint,  OSD  directed  the 
establishment  of  the  ICA  program.  The  con¬ 
cept  was  that  cost  estimators,  outside  the  influ¬ 
ence  of  program  advocacy,  would  develop  cost 
estimates  that  more  accurately  portrayed  the 
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challenges,  risks,  and  costs  associated  with  the 
development  and  production  cf  advanced 
weapon  systems. 

In  addition,  the  requirement  for  inde¬ 
pendent  cost  estimates  has  been  contained  in 
public  law  as  follows: 

1984 Authorization  Act-.  “...The  Secre¬ 
tary  of  Defense  may  not  approve  the  full-scale 
engineering  development  or  the  production 
and  deployment  of  a  major  defense  acquisition 
program  unless  an  independent  estimate  of  the 
cost  of  the  program  has  been  submitted  to  the 
Secretary  of  Defense...” 

1985  Authorization  Act :  “...Not  later 
than  May  1,  1985,  the  Secretary  of  Defense 
shall  submit  to  the  Committees  on  Armed 
Services  of  the  Senate  and  House  of  Repre¬ 
sentatives  a  report  on  the  continued  use  of  the 
independent  cost  estimates  in  the  planning, 
programming,  budgeting,  and  selection  proc¬ 
ess  for  major  defense  acquisition  in  the  Depart¬ 
ment  of  Defense. 

Department  of  Defense  Directive 
5000.4  establishes  the  OSD  Cost  Analysis  Im- 
'  provement  Group  (CAIG).  This  directive  dis¬ 
cusses  how  cost  estimates  are  to  be  presented 
to  the  OSD  CAIG,  and  specifies  its  member¬ 
ship.  The  OSD  CAIG  acts  as  the  principal  advi¬ 
sor  to  Undersecretary  of  Defense  Research 
and  Engineering  (USDR&E),  and  the  Defense 
Advisory  Board  on  matters  relating  to  pro¬ 
gram  cost.  The  OSD  CAIG  is  required  to  re¬ 
view  and  recommend  action  on  cost  estimates 


for  all  major  systems.  The  same  directive  iden¬ 
tifies  two  types  of  estimates: 

•  Independent  Cost  Estimates 
(ICE) 

•  Program  Office  Cost  Esti¬ 
mates  (PCE). 

An  IC  A  starts  with  an  Independent  Cost 
Estimate  (ICE)  prepared  in  response  to  the 
DoD  ICA  program.  Developing  an  ICA  basi¬ 
cally  entails  the  same  procedures,  methodolo¬ 
gies,  and  techniques  that  would  be  employed 
to  accomplish  any  other  weapon  system  cost 
estimate.  However,  ideally  the  ICA  should  se¬ 
lect  methodologies  and  techniques  different 
from  those  underlying  the  Program  Office  Cost 
Estimate  (PCE).  In  addition,  an  ICA  should  in¬ 
clude  a  detailed  comparison  and  explanation  of 
differences  between  the  ICA  and  PCE. 

The  key  aspect  of  the  independent  cost 
estimate  is  that  it  is  developed  in  organiza¬ 
tional  channels  separate  and  independent  from 
the  program  office.  This  helps  it  serve  as  an 
analytical  tool  to  validate  or  cross-check  pro¬ 
gram  management  office  developed  cost  esti¬ 
mates.  This  second  opinion  helps  avoid  the  risk 
that  some  significant  costs  have  been  over¬ 
looked  or  that  PMO  program  advocacy  has  re¬ 
sulted  in  low  estimates  which  could  place  the 
success  of  the  program  at  risk. 

To  the  extent  that  those  preparing  inde¬ 
pendent  cost  estimates  are  advised  and  sup¬ 
ported  by  a  technical  staff  independent  of  the 
program  office  staff,  some  independent  assess- 
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ment  of  technical  risks  may  also  be  accom¬ 
plished  during  preparation  of  an  independent 
cost  estimate. 

5.14  RISK  HANDLING  TECHNIQUES 

Handling  technique  classifications  were 
covered  in  Section  4.5.  The  possibilities  for 
dealing  with  risk  are  as  varied  as  potential 
sources  of  risk.  It  would  be  impossible  to  dis¬ 
cuss  each  technique  without  first  describing  the 
complete  circumstances  under  which  it  is  ap¬ 
propriate.  The  key  to  developing  an  appropri¬ 
ate  handling  of  any  risk  lies  in  the  proper 
execution  of  the  risk  planning,  risk  assessment, 
and  risk  analysis  functions.  If  these  are  done 
properly,  the  impacts  of  potential  actions  will 
be  clearly  understood  and  will  lead  to  the  best 
possible  risk  handling  action. 

The  following  Table  5.14-1  shows  some 
of  the  typical  activities  that  should  be  per¬ 
formed  in  each  phase  of  the  development  cycle. 
Clearly,  management  actions  to  reduce  risk 
should  be  aimed  at  performing  quality  work  on 
each  of  these  items.  One  of  the  primary  reasons 
that  this  structural  approach  to  acquisition  ex¬ 
ists  is  to  reduce  the  risks  of  buying  a  piece  of 
equipment  that  does  not  meet  the  need,  does 
not  live  up  to  the  performance  requirement,  is 
too  costly,  or  is  too  late. 


5.15  CHAPTER  5  KEY  POINTS 


•  Risk  Management  tech¬ 
niques  can  apply  to  multiple 
parts  of  the  risk  manage¬ 
ment  process 

•  Some  techniques  special¬ 
ize  in  one  aspect  of  risk 

•  Techniques  should  be  se¬ 
lected  based  on  program 
requirements  (Chapter  6 
provides  detail) 

•  No  technique  will  give  you 
a  choice  or  management 
actions 

•  Management  actions  are 
limited  only  by  the  inge¬ 
nuity  of  the  program  man¬ 
ager. 
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Table  5.14-1  Typical  Activities  by  Program  Phase 


CONCEPT  EXPLORATION 

DEMONSTRATION/VALIDATION 

Identify  Manufacturing  Technology  Needs 

Examine  Producibility  of  Competitive  Designs 

Identify  Critical  Materials 

Prepare  for  Production  Readiness  Review 

Evaluate  Risk  of  Manufacturing  Alternatives 

Prepare  Initial  Manufacturing  Plan 

Perform  Industrial  Base  Analysis 

Evaluate  Long  Lead  Requirements 

Determine  Contract  Requirements  for  Dem/Val 

Determine  Need  for  LRIP 

Define  System  Level  Logistics  Requirements  (ILSP) 

Prepare  Initial  Production  Cost  Estimate 

Perform  Initial  Facility  Planning 

Determine  Contract  Requirements  for  FSD 

Estimate  Life  Cycle  Cost  Performance  Goals 

Establish  Readiness  and  Supportability  Objectives 

Develop  System  Specifications 

Prepare  for  Development  and  Operational  Testing 

Conceptualize  T&E  Program  (TEMP) 

Determine  Acquisition  Strategy 

FULL  SCALE  DEVELOPMENT 

PRODUCTION 

Define  Required  Manufacturing  Resources 

Ensure  Facilities  are  In  Place 

Prepare  Manufacturing  Cost  Estimates 

Examine  Use  of  Warranties 

Perform  Production  Risk  Assessment 

Determine  Acquisition  Strategy 

Accomplish  Production  Planning 

Exauine  Use  of  Second  Source 

Assess  Long  Lead  Material  Requirements 

Integrate  Spares  Production 

Perform  Producibility  Studies 

Perform  Fielding  Analysis 

Complete  Manufacturing  Plan 

Perform  Contractor  Production  Surveillance 

Accomplish  Development  and  Operational  Testing 

Execute  Product  Improvement  Initiatives 

Perform  Production  Readiness  Reviews 

Determine  Contract  Requirements  for  Production 

Determine  Quality  &  Peri'orr.iar Controls  for  Production 
Evaluate  Impact  of  Engineering  Changes  on  LCC 

Prepare  for  Transition  to  Production 

Implement  Value  Engineering 
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Chapter  6 

RISK  MANAGEMENT  IMPLEMENTATION 


6.1  INTRODUCTION 

While  the  concepts  and  techniques  of 
risk  planning,  assessment,  analysis,  and  han¬ 
dling  are  complex,  the  greater  challenge  is  in 
the  actual  implementation  of  the  risk  manage¬ 
ment  process.  Program  managers  and  PMOs 
are  almost  categorically  overcommitted  and 
overextended.  In  a  recent  Risk  Analysis  and 
Management  survey  of  DoD  Program  Man¬ 
agement  Offices  (PMO),  allocating  the  re¬ 
sources  to  implement  an  effective  risk 
management  program  was  a  significant  and 
frequently  reported  problem.  Over  50  percent 
of  the  PMOs  responded  that  inadequate  pro¬ 
gram  staffing  was  a  major  risk  area  in  and  of  it¬ 
self.  The  view  of  risk  management 
implementation  as  an  additional  requirement 
levied  on  the  program  team  can  appear  as  an 
overwhelming  task.  In  actuality,  risk  manage¬ 
ment  is  an  integral  part  of  program  manage¬ 
ment,  not  an  additional  analysis  task.  Risk 
management  affects  each  of  the  classic  ele¬ 


ments  of  management;  planning,  organizing, 
directing,  and  controlling.  Risk  management 
plays  an  important  role  in  the  decision  making 
process.  In  essence,  risk  management  is  a  sub¬ 
set  of  sound  program  management  and  while 
the  level  or  activity  may  vary,  risk  management 
should  be  viewed  as  an  on-going  process  ver¬ 
sus  a  one  time  exercise,  as  illustrated  in  Figure 
6.1-1. 

Risk  Management  implementation 
means  the  incorporation  of  the  risk  manage¬ 
ment  concepts  and  techniques  into  the  pro¬ 
gram  management  process,  not  simply  the 
manipulation  of  a  certain  model.  To  this  end. 
this  chapter  provides  guidance  for: 

•  Organizing  for  Risk 
Management 

•  Technique  Selection 

•  Risk  Management 
Resource  Allocation 

•  Communicating  Risk 

•  Developing  a  Risk 
Management  Capability. 
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Figure  6.1-1  Risk  Management  as  a  Process 


As  with  all  efforts,  successful  risk  man¬ 
agement  implementation  is  a  function  of  the 
organization’s  understanding  and  commitment 
to  meet  the  challenge. 

6.2  ORGANIZING  FOR  RISK 
MANAGEMENT 

The  program  manager  is  ultimately  re¬ 
sponsible  for  the  implementation  of  risk  man¬ 
agement.  The  program  manager  establishes 
goals  for  the  risk  management  effort  and  allo¬ 
cates  the  resources  to  accomplish  these  objec¬ 
tives.  While  the  program  manager  must 
oversee  this  process,  risk  management  activi¬ 
ties  do  not  reside  solely  with  the  program  man¬ 


ager.  Risk  typically  manifests  itself  in  the 
functional  analysis  and  decision  making  proc¬ 
ess.  Figure  6.2-1  depicts  a  sample  of  functional 
analysis  which  often  involves  the  complex  in¬ 
terplay  of  technical,  programmatic,  suppor- 
lability,  cost,  and  schedule  risk.  Functional 
managers  must  understand  the  implications 
risk  has  in  each  of  their  respective  disciplines. 
Risk  management  is  a  significant  responsibility 
in  each  of  the  functional  manager’s  jobs.  The 
program  manager’s  role  is  to  provide  the  moti¬ 
vation  and  structure  to  effectively  manage  risk. 
The  program  manager  should  promote  the 
continual  interaction  between  team  members 
for  communication  concerning  risk  manage¬ 
ment. 
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Figure  6.2-1  Functional  Roles  In  Risk  Management 


While  it  is  clear  that  risk  management  is  Either  approach  could  be  defended  for 

a  team  function,  it  is  not  obvious  how  to  best  organizing  for  risk  management.Though  dif- 

organize  that  team  to  execute  the  process.  A  ferent,  three  basic  themes  appear  as  guidelines 

survey  of  DoD  Program  Management  Offices  for  incorporating  risk  management  into  the 

risk  management  activities  revealed  two  basic  program  management  process.  First,  the  pro¬ 
approaches  to  organizing  for  risk  management  gram  manager  is  ultimately  responsible,  as 

that  the  respondents  felt  were  successful.  One  with  all  aspects  of  the  program,  for  the  plan- 

group  of  PMOs  designated  specific  positions  to  nipg,  allocation  of  resources,  and  execution  of 

conduct  the  program’s  risk  management  ef-  risk  management.  Second,  risk  management  is 

forts.  The  number  of  people  allocated  varied  a  team  function.  Each  functional  manager 

by  the  size  of  the  PMO  and  the  risk  manage-  plays  an  important  role  in  the  identification, 

ment  techniques  being  used.  The  other  PMOs  analysis,  and  handling  of  risk.  Third,  risk  man- 

felt  that  risk  management  was  such  an  integral  agement  activities  and  responsibilities  must  be 

part  of  engineering  and  management  that  sepa-  specific  and  assigned  to  individuals.  Actions 

rate  personnel  weie  not  designated  to  manage  and  responsibilities  assigned  to  groups  are,  in 

risk  and  adequate  consideration  of  risk  was  be-  effect,  not  assigned.  Whether  risk  management 

ing  accomplished  as  a  normal  part  of  their  jobs.  is  a  full  time  job  or  an  integral  part  of  a  team 
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member’s  job,  risk  management  actions  should 
be  explicit  and  assignments  clear. 

63  RISK  ASSESSMENT  AND 

ANALYSIS  TECHNIQUE 

SELECTION 

Establishing  objectives  and  allocating 
resources  to  accomplish  those  objectives  is  a 
primary  function  of  the  program  manager.  As 
depicted  in  Figure  6.3-1,  this  function  is  the 
basis  for  risk  planning,  the  first  step  in  the  risk 
management  process.  At  the  heart  of  this  plan¬ 
ning  effort  is  the  selection  of  the  most  appro¬ 
priate  risk  assessment  and  analysis  techniques 
for  the  program.  Selection  of  risk  assessment 
and  analysis  techniques  is  the  subject  of  this 
section.  The  technique  selected  shapes  the  na¬ 
ture  of  the  risk  management  effort  and  should 
be  directed  towards  providing  the  information 
necessary  to  meet  the  risk  management  objec¬ 
tives  within  the  resource  constraints  of  the 
PMO. 

This  discussion  focuses  on  those  risk 
management  techniques  described  in  Chapter 
5  that  deal  specifically  with  risk  assessment  and 
risk  analysis.  Figure  6.3-2  illustrates  the  idea 
that  risk  management  techniques  can  be 
loosely  categorized  by  the  primary  purpose 
they  serve  in  the  risk  management  process. 
Generally,  the  risk  identification  and  quantifi¬ 
cation  techniques  can  support  a  variety  of  risk 
assessment  and  analysis  approaches.  Expert  in¬ 
terviewing  techniques  are  equally  applicable 
for  obtaining  information  in  doing  a  network 
analysis,  decision  analysis,  or  developing  a  life 


cycle  cost  estimate.  Similarly,  the  risk  handling 
technique  decision  is  not  a  function  of  the  risk 
assessment  methodology  employed.  Insights 
from  the  implementation  of  a  Watchlist,  for  ex¬ 
ample,  support  the  use  of  several  approaches  to 
risk  handling.  The  type  and  timing  of  informa¬ 
tion  needed  for  specific  decision  making  appli¬ 
cations,  however,  form  the  guidelines  and 
constraints  for  the  selection  of  the  appropriate 
risk  assessment  and  analysis  technique.  The 
following  section  provides  guidance  and  a  gen¬ 
eral  framework  of  comparison  for  the  selection 
of  effective  techniques  for  risk  assessment  and 
analysis.  The  answer  is  not  the  same  for  each 
program,  nor  does  the  answer  necessarily  stay 
the  same  for  the  life  of  a  single  program.  As  il¬ 
lustrated  by  Figure  6.3-3  the  nature  and  level 
of  risk  management  activity  varies  through  the 
acquisition  life  cycle  of  a  program.  The  risk  as¬ 
sessment  and  analysis  techniques  that  are  ef¬ 
fective  between  Milestone  I  and  II,  when  a  firm 
technical  baseline  is  not  yet  established,  may 
be  inappropriate  in  the  late  production  phase 
of  the  program.  The  resources  required  for 
risk  management  activity  varies  with  the  tech¬ 
niques,  and  the  techniques  used  are  largely  de¬ 
pendent  op  the  objectives  of  the  risk 
management  process. 


63.1  Technique  Selection  Criteria 

A  variety  of  interrelated  factors  affect 
the  selection  of  a  technique.  The  current  acqui¬ 
sition  phase,  size,  priority,  and  complexity  of  a 
program  all  affect  the  type  of  information  and 
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Figure  6 .3-1 
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analysis  required  to  deal  with  risk.  A  key  con¬ 
sideration  and  often  a  major  constraint  is  the 
availability  and  capability  of  resources  to  de¬ 
vote  to  risk  management.  The  pressure  to  do 
more  with  less  is  a  constant  and  pervasive  con¬ 
dition  in  PMOs.  Often,  organizations  also  have 
policies  or  directives  which  require  the  use  of 
one  or  another  risk  assessment  or  analysis  tech¬ 
nique.  The  objectives  of  the  risk  management 
effort  tie  these  considerations  together  and 
balances  their  influence  in  the  selection  of  an 
appropriate  technique. 

These  factors  can  be  aggregated  into 
three  categories  for  the  purposes  of  discussion 
and  technique  evaluation  and  selection  (Figure 
6.3-4). 

•  Resource  requirements 

•  Application 

•  Output. 


Serious  consideration  of  the  resources 
required  to  execute  a  particular  technique  was 
a  recurring  theme  in  the  responses  to  the  ques¬ 
tion  of  why  a  particular  technique  was  used  by 
the  PMO  in  the  DSMC  Risk  Management  sur¬ 
vey  results.  The  second  criteria  is  the  applica¬ 
tion  or  the  decision  making  process  to  which 
the  risk  assessment  or  analysis  is  targeted.  The 
specific  purpose  or  application  of  the  risk  infor¬ 
mation  obtained  varies  and  changes.  Different 
techniques  better  support  different  applica¬ 
tions,  thus  the  application  in  which  the  risk  in¬ 
formation  is  used  is  another  key  criteria 
category.  The  third  criteria  is  the  actual  output 
from  the  risk  assessment  and  analysis  tech¬ 
nique.  The  accuracy,  level  of  detail,  and  utility 
of  the  technique  output  should  best  match  the 
required  information  for  risk  management  de¬ 
cision  making. 
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The  criteria  discussed  are  not  alien- 
compassing  and  clearly  other  circumstances 
can  influence  the  selection  of  a  risk  technique. 

However,  these  criteria  will  provide  a 
comparative  yardstick  for  evaluation  and  a 
framework  for  an  educated  decision  to  select  a 
technique  and  implement  it  in  the  risk  manage¬ 
ment  process. 

This  section  discusses  several  criteria 
that  can  be  used  to  evaluate  which  risk  assess¬ 
ment  and  analysis  technique  fits  the  require¬ 
ments  and  constraints  of  a  program's  risk 
management  effort.  Each  of  the  major  tech¬ 
niques  is  then  evaluated  against  these  criteria 
and  a  general  approach  to  technique  selection 
is  discussed.  The  intent  is  not  to  make  tech¬ 
nique  selection  automatic,  but  to  help  point  out 
the  advantages  and  disadvantages  of  different 
techniques  in  different  circumstances. 

63.1.1  Resource  Requirements 

What  resources  a  particular  technique 
requires  is  often  the  dominant  consideration  in 
the  selection  process. 

The  greatest  utility  witn  the  least  time, 
money,  and  .manpower  expended  is  always  the 
sought  after  objective.  The  resource  require¬ 
ments  of  the  various  risk  assessment  and  analy¬ 
sis  techniques  are  compared  using  the 
following  five  factors: 

•  Cost 

•  Hardware/Software 
Tools  needed/available 

•  Time  to  implement 


•  Ease  of  Use 

*  PM  Commitment. 

The  cost  identified  is  a  rough  approximation  of 
the  labor  required  (in  man-months)  to  conduct 
or  initially  set  up  the  risk  assessment  and  analy¬ 
sis.  Several  techniques  are  maintained  over  an 
extended  period  of  time.  The  maintenance  of 
these  techniques  is  not  considered  in  the  com¬ 
parative  cost  figures.  Obviously,  actual  costs 
vary  considerably  depending  on  the  size  and 
complexity  of  the  program  and  the  scope  of  the 
assessment  and  analysis.  Thus  the  costs  de¬ 
picted  are  for  comparative  purposes.  The  hard¬ 
ware/software  factor  simply  indicates  whether 
or  not  (Yes;  No)  special  hardware  or  software 
analysis  packages  are  typically  needed  to  use 
the  technique. 

Time  indicates  the  duration  of  time  (in 
months)  needed  to  implement  the  individual 
technique.  Again,  in  those  techniques  requir¬ 
ing  continuing  maintenance,  only  the  initial 
time  to  implement  is  considered. 

Ease  of  use  is  a  subjective  assessment  of 
the  relative  difficulty  in  implementing  each 
technique.  A  three  point  scale  of  E  (Easy):  M 
(Moderate);  D  (Difficult)  is  used  to  rate  each 
technique. 

The  last  resource  requirement  factor 
examined  and  rated  is  the  program  manager’s 
time  commitment  to  successfully  implement 
the  technique.  Obviously  a  technique  which  re¬ 
quires  intensive  and  continual  involvement  of 
the  program  manager  would  be  difficult  to  im- 
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plement.  A  three  point  scale  of  S  (Slight);  M 
(Moderate);  H  (Heavy)  is  used  to  rate  each 
technique. 

Evaluation  of  the  techniques  against 
each  of  these  factors  involved  in  the  resource 
requirements  criteria  is  presented  in  Section 
6.3.2. 

6.3.1.2  Technique  Application 

The  following  applications  are  defined 
here  and  matched  against  the  capabilities  of 
the  techniques  evaluated  in  Section  6.3.2,  using 
a  three  point  scale  of  H  (High);  M  (Medium);  L 
(Low). 

•  Program  Status  Reporting 

•  Major  Planning  Decisions 

•  Acquisition  Strategy  Selec¬ 
tion 

•  DAB  Milestone  Prepara¬ 
tion 

•  Design  Guidance 

•  Source  Selection 

•  POM/Budget  Submittal. 

Program  Status  Reporting  refers  to  the 
monitoring  of  plans,  costs,  and  schedules  to 
ensure  that  standards  are  met  and  problems 
identified  for  timely  corrective  action. 

Major  Planning  Decisions  refers  to  ma¬ 
jor  decisions  to  which  a  program  manager  may 
be  willing  to  invest  significant  resources  and 
personal  attention. 

Acquisition  Strategy  Selection  typically 
occurs  several  times  throughout  the  life  of  a 
program.  Risk  assessment  and  analysis  provide 


key  information  relevant  to  the  tradeoffs  and 
cost  benefit  analysis  of  contract  type  selection, 
warranty  structuring,  etc. 

The  application  of  risk  assessment  and 
analysis  in  the  Defense  Acquisition  Board 
(DAB)  Milestone  Preparation  is  very  direct  and 
important.  The  objective  of  the  DAB  is  to  in¬ 
sure  the  major  weapon  systems  planning  has 
been  comprehensive  and  the  system  is  ready  to 
proceed  into  the  next  acquisition  phase. 

The  next  application  category  consid¬ 
ered  in  evaluating  the  techniques  is  Design 
Guidance.  From  the  consideration  of  technol¬ 
ogy  alternatives  for  major  weapon  systems  to 
the  choices  of  components,  each  alternative 
represents  a  collection  of  large  uncertainties  of 
cost,  schedule,  and  technical  performance.  In 
each  situation,  the  program  manager  will  want 
to  understand  how  the  uncertainties  relate  to 
one  another  and  how  the  alternatives  compare. 

Source  Selection  evaluations  frequently 
involve  the  consideration  of  risk  as  a  determi¬ 
nant  of  selection.  A  quantified  risk  manage¬ 
ment  effort  provides  the  information  to 
substantiate  an  evaluation.  Source  selection  is 
a  prime  application  for  risk  assessment  and 
analysis.  The  typical  short  duration  of  source 
selections  and  their  necessary  restrictive  nature 
place  constraints  on  the  type  of  technique  used 
and  the  level  of  detail  that  can  be  successfully 
pursued. 

POM! Budget  Submittal  is  an  obvious 
periodic  application  category.  The  basic  deci- 
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sion  of  what  funds  are  required  to  accomplish 
the  program  direction  is  an  exercise  in  under¬ 
standing  and  evaluating  the  interplay  of  techni¬ 
cal,  supportability,  programmatic,  cost,  and 
schedule  risk  factors. 

63.13  Technique  Output 

The  third  group  of  factors  examined  to 
compare  and  evaluate  risk  assessment  and 
analysis  techniques  consider  the  output  of  the 
risk  effort  in  terms  of: 

•  Accuracy 

•  Level  of  detail 

•  Utility. 

These  factors  are  defined  here  and  matched 
against  the  capabilities  of  the  techniques  evalu¬ 
ated  in  Section  6.3.2  using  a  three  point  scale  of 
H  (High);  M  (Medium);  L  (Low). 

Accuracy  deals  with  the  basic  theoreti¬ 
cal  soundness  of  the  technique  and  the  neces¬ 
sity  for  weakening  assumptions  which  may 
dilute  the  value  of  the  information  obtained  in 
the  analysis.  Most  techniques  present  an  obvi¬ 
ous  trade-off  between  ease  of  use  or  time  com¬ 
mitment  and  the  final  accuracy  of  the  analysis 
results. 

Level  of  Detail  is  concerned  with  the 
output  contents  capability  to  provide  more  de¬ 
tailed  insights  into  cost,  schedule,  and  technical 
risks.  Techniques  and  how  they  are  applied 
vary  in  the  breadth,  depth,  and  understanding 
that  the  output  contents  provide. 


Utility  is  a  subjective  factor  which  rates 
the  output  in  a  general  context  of  its  usefulness 
to  the  PMOs.  Both  the  effort  involved  in  the 
risk  assessment  and  analysis  and  the  end  value 
of  information  is  considered. 

The  ratings  are  obviously  subjective, 
but  their  discussion  brings  out  important  con¬ 
siderations  in  choosing  a  risk  assessment  or 
analysis  technique.  The  feedback  from  the 
DSMC  Risk  Management  survey  has  been  util¬ 
ized  in  the  rating  and  comparison  of  the  indi¬ 
vidual  techniques. 

63.2  Technique  Evaluation 

This  section  rates  and  discusses  each  of 
the  risk  analysis  and  assessment  techniques  in 
the  context  of  the  previously  defined  selection 
criteria.  This  presentation  will  not  make  the  se¬ 
lection  of  a  risk  technique  automatic.  Its  inten¬ 
tion  is  to  provide  the  PMOs  with  an  informed 
perspective  to  evaluate  and  choose  an  ap¬ 
proach  that  is  suited  to  meet  the  objectives  of 
the  risk  management  effort  within  the  ever  pre¬ 
sent  resource  constraints  of  a  program.  Table 

6.3- 1  is  a  matrix  of  the  results  of  evaluating 
each  technique  against  the  previously  defined 
selection  criteria. 

It  is  more  important  to  note  that  some 
techniques  have  more  applicability  to  specific 
program  phases.  Likewise,  each  technique 
yields  different  information  than  others.  Table 

6.3- 2  summarizes  the  technique  applicability 
for  each  program  phase  and  addresses  the  type 
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Table  6 3-2  Program  Phase/Technique  Application 


PROGRAM  PHASE 

INFORMATION  YIELD 

CE 

D/V 

FSD 

PROD 

TECH 

PROG 

SUP 

COST 

SCHED 

EXPERT  INTERVIEWS 

+ 

+ 

+ 

+ 

+ 

B 

+ 

0 

0 

ANALAQOUS  SYSTEMS 

0 

+ 

+ 

+ 

0 

0 

+ 

0 

PLAN  EVALUATION 

- 

0 

+ 

+ 

+ 

0 

+ 

- 

- 

TRANSITION  TEMPLATES/ 

0 

+ 

+ 

+ 

+ 

0 

+ 

_ 

LESSONS  LEARNED  STUDIES 

NETWORKS 

- 

+ 

+ 

0 

+ 

0 

+ 

+ 

+ 

DECISION  ANALYSIS 

+ 

+ 

0 

0 

0 

0 

0 

0 

0 

ESTIMATING  RELATIONSHIPS 

- 

- 

- 

0 

- 

- 

- 

+ 

- 

RISK  FACTORS 

- 

0 

+ 

+ 

- 

- 

- 

+ 

- 

LIFE  CYCLE  COST  MODELS 

- 

0 

0 

+ 

- 

- 

+ 

+ 

- 

COST  RISK  SIMULATIONS 

- 

B 

+ 

+ 

B 

- 

- 

+ 

- 

PERFORMANCE  TRACKING 

0 

D 

+ 

H 

0 

+ 

+ 

+ 

of  information  likely  to  be  received.  This  table 
shows  “general”  guidelines.  There  hcve  been, 
and  will  continue  to  be  specific  applications 
that  are/will  be  exceptions  to  the  guidance  rep¬ 
resented  in  this  table.  Technique  selection 
should  not  be  based  solely  on  program  phase. 
The  type  of  information  desired  as  a  result  of 
the  execution  of  a  particular  technique  must 
also  be  considered.  For  example,  while  net¬ 
works  are  not  the  optimum  risk  analysis  tool  in 


-  ■  Relatively  weak  application/information 
0  s  Average  application/information 
+  s  Relatively  strong  application/information 

production,  they  may  be  desirable  because  of 
their  value  in  planning  and  control  while  tran¬ 
sitioning  from  development  to  production. 
Similarly,  networks  may  serve  a  somewhat  dif¬ 
ferent  purpose  in  the  different  phases. 

A  more  thorough  discussion  of  each 
technique  was  presented  in  Chapter  5.  This 
evaluation  will  summarize  the  key  characteris¬ 
tics  to  be  considered  in  making  the  proper  se¬ 
lection  of  a  technique. 
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63.2.1  Selection  Criteria  for  Network 
Analysis  Technique 

Resource  Requirement:  The  resources 
required  to  apply  the  network  analysis  tech¬ 
nique  is  dependent  on  a  few  factors.  One  of 
which  is  whether  or  not  the  networks  already 
exist  for  the  program.  If  so,  they  can  be  utilized 
for  the  risk  models  and  much  labor  can  be 
saved.  The  scope  of  the  program  and  the  level 
of  detail  being  modeled  also  impact  the  re¬ 
source  requirements  (with  the  larger  scope  and 
greater  detail  requiring  significantly  more  re¬ 
sources).  When  doing  network  risk  analysis, 
special  software  is  required.  Also,  if  plots  of 
the  networks  are  desired,  plotting  equipment 
will  be  needed.  Since  the  process  of  building 
the  networks,  capturing  expert  judgment  and 
understanding  the  software  are  not  simple 
tasks,  ease  of  use  would  be  rated  as  low.  PM 
backing  is  mandatory  for  successful  network 
analysis  because  of  the  resources  required,  and 
the  degree  of  difficulty  associated  with  the 
process.  Although  the  PM’s  personal  involve¬ 
ment  is  slight  to  moderate,  the  members  of  the 
program  team  must  be  convinced  of  the  man¬ 
ager’s  commitment  to  the  task. 

Application:  Networks  have  a  high  de¬ 
gree  of  utility  as  discussed  in  Section  5.8,  there¬ 
fore  all  of  the  applications  listed  are  relevant. 

Output :  With  respect  to  output,  the  ac¬ 
curacy  of  the  analysis  is  a  function  of  the  valid¬ 
ity  of  the  network  itself  and  the  PDFs 
constructed  for  each  activity.  The  level  of  detail 
is  determined  by  management  so  it  can  be  low, 
medium,  or  high.  The  utility  of  the  networks  is 
generally  high,  if  for  no  other  reason  than  it 


forces  managers  to  detail  plan  before  the  exe¬ 
cution  of  a  program. 

63.2.2  Selection  Criteria  for 

Decision  Analysis  Technique 

Resource  Requirement:  The  decision 
analysis  method  is  a  much  simpler  technique 
than  network  analysis.  Because  of  this,  the  re¬ 
sources  required  are  significantly  less.  PM 
commitment,  while  required,  does  not  need  to 
be  as  high  as  with  network  analysis. 

Application:  As  with  network  analysis, 
decision  analysis  lends  itself  well  to  all  of  the 
potential  applications  listed. 

Output:  If  the  program  can  be  accu¬ 
rately  modeled,  the  output  will  be  accurate. 
The  level  of  detail  is  specified  by  the  program 
manager  predicated  on  what  he  deems  neces¬ 
sary.  The  utility  of  decision  analysis  is  not  as 
high  as  network  analysis  because  it  does  not 
provide  the  same  diversity  of  output  or  address 
the  myriad  of  questions  that  network  analysis 
does. 

63.23  Selection  Criteria  for 

Estimating  Relationships 
Technique 

General:  The  estimating  relationship 
method  is  not  well  understood  by  many.  Many 
PMO  survey  responses  indicated  they  had  used 
the  technique  when  they  had  really  used 
parametric  cost  estimating  methods  for  some 
or  all  of  the  program  cost  estimates.  Such 
analysis  is  more  accurately  described  as  all  or 
part  of  a  life  cycle  cost  analysis.  The  estimating 
relationship  method  is  defined  by  the  use  of 


6-13 


parametric  estimating  methods  to  estimate  risk 
or  management  reserve  fund  requirements. 
There  are  currently  very  few  parametric  cost 
models  available  with  which  to  do  this. 

Resource  Requirements’.  The  primary  re¬ 
quirement  is  the  availability  of  a  parametric 
cost  model  specifically  designed  to  estimate 
management  reserve  or  risk  funds  as  a  function 
of  one  or  more  program  parameters.  If  such  a 
model  is  not  available,  one  to  three  months 
may  be  required  to  develop  one.  If  the  required 
historical  data  is  not  available,  it  may  be  impos¬ 
sible  to  develop  the  required  cost  model.  If  a 
satisfactory  model  is  available,  it  generally 
takes  only  a  few  days  to  use  it.  However,  the 
program  manager  must  support  its  use  so  key 
program  personnel  will  provide  the  cost  analyst 
with  timely  judgments  or  information  needed 
to  input  the  model.  The  model  equations  are 
usually  so  simple  that  a  handheld  calculator 
can  be  used  to  compute  required  management 
reserve  fund  requirements. 

Application :  The  primary  use  of  this 
method  is  to  compute  the  management  reserve 
or  risk  funds  to  be  included  in  POM  and  BES 
funding  requirements.  It  has  little  or  no  use  for 
other  applications.  This  is  a  very  easy  technique 
to  recompute  and  update  as  the  program  pro¬ 
gresses  over  time  and  either  the  level  of  risk  or 
the  basic  program  cost  estimate  changes. 

Output :  The  estimating  relationship 
method  output  is  generally  a  percent  value. 
This  value  multiplied  by  the  basic  program  cost 
estimate  provides  an  estimate  of  the  manage¬ 
ment  reserve  or  risk  funds  that  must  be  added 


to  the  basic  cost  estimate  to  assure  the  budget 
includes  adequate  funds  to  cover  the  added 
costs  that  wi'l  be  incurred  because  of  the  risks 
associated  with  the  program.  The  accuracy  of 
this  method  is  considered  low  primarily  be¬ 
cause  the  historical  data  bases  upon  which  such 
models  are  based  are  small,  and  it  is  often  hard 
to  accurately  define  what  funds  were  spent  to 
address  risk  on  past  programs.  This  method 
provides  little  or  no  detail  with  respect  to  which 
parts  of  the  program  are  more  risky,  and  there¬ 
fore,  more  likely  to  require  additional  funding. 
Since  there  are  so  few  models  of  this  type  avail¬ 
able,  and  even  their  use  are  subject  to  question, 
the  overall  utility  of  this  method  across  the 
DoD  must  be  considered  low. 

63.2.4  Selection  Criteria  For 
Transition  Templates 
Technique 

Resource  Requirements :  This  technique 
requires  little  additional  resources  above  what 
is  normally  necessary  to  properly  manage  a 
program.  There  are  no  special  hardware/soft¬ 
ware  requirements  and  the  technique  is  easy  to 
use.  It  does  require  discipline  on  the  part  of  the 
program  management  office  to  regularly  re¬ 
view  and  compare  progress  against  each  of  the 
template  areas. 

Application :  The  Transition  Templates 
can  be  used  in  most  of  the  application  catego¬ 
ries  in  Table  6.3-1.  The  technique  is  only  indi¬ 
rectly  useful  in  the  POM/Budget  category 
because  it  deals  with  preventive  technical  as¬ 
pects  rather  than  cost  issues.  It  can,  however, 
provide  insight  into  the  driving  forces  behind 
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cost  and  a  contractor’s  methodology  for  man¬ 
aging  a  program  in  a  source  selection  situation. 

Output :  If  the  user  properly  documents 
the  results  of  his  analysis,  the  output  will  pro¬ 
vide  a  traceable  management  checklist  that  can 
be  used  to  make  sound  decisions  on  technical 
issues.  Again,  discipline  is  a  key  issue  in  deter¬ 
mining  the  usefulness  of  the  output.  If  each  of 
the  templates  is  examined  in  detail,  the  user 
will  have  a  firm  understanding  of  the  technical 
risks  faced  in  the  program.  Skipping  templates 
t  jcause  there  is  no  “apparent”  risk  may  save 
time,  but  may  also  miss  key  problem  areas. 

6 .3.2.5  Selection  Criteria  for 
Life  Cycle  Cost 
Analysis  Technique 

General:  Life  cycle  cost  analysis  has 
been  used  widely  in  the  DoD  in  recent  years  as 
a  result  of  growing  concern  about  rapidly  in¬ 
creasing  operating  and  support  costs.  Since 
economic  considerations  at  e  an  integral  part  of 
engineering,  life  cycle  costs  can  be  an  impor¬ 
tant  design  consideration.  There  are  both 
PMO  unique  and  general  purpose  life  cycle 
cost  models  widely  used.  The  availability  of  the 
electronic  spreadsheet  has  greatly  facilitated 
the  development  and  use  of  life  cycle  cost  mod¬ 
els  by  PMO  personnel. 

Resource  Requirements :  Performing  a 
life  cycle  cost  analysis,  even  one  involving  many 
estimates  for  different  scenarios  or  sets  of  as¬ 
sumptions  can  be  done  relatively  easily  and 
quickly,  if  the  model  is  already  available.  For 


large  and  complex  programs,  the  input  data 
collection  process  may  involve  many  sources. 
Even  the  most  complex  life  cycle  cost  models 
can  be  run  on  computers  of  the  size  avr '?.ble  to 
most  PMOs.  If  a  PMO  unique  model  is  not 
available  and  must  be  developed,  the  resource 
requirements  to  do  this  can  be  greatly  reduced 
if  a  demonstrated  model  for  a  similar  system 
can  be  tailored  to  the  PMOs  needs. 

Application :  Since  cost  is  an  important 
management  consideration,  the  results  of  life 
cycle  cost  analysis  are  applicable  to  many  PMO 
decisions  and  activity  areas.  Once  a  PMO  com¬ 
puterized  life  cycle  cost  analysis  capability  has 
been  developed,  it  is  of  significant  value  when¬ 
ever  a  quick  assessment  is  needed  of  the  cost 
implications  of  design,  production  rate,  or 
other  program  changes. 

Output :  The  overall  accuracy  of  life  cy¬ 
cle  cost  analysis  is  medium.  Usually  such  esti¬ 
mates  can  be  improved  if  significant  additional 
time  is  taken  to  have  the  prime  contractor  take 
a  more  detailed  look  at  how  changes  would  im¬ 
pact  a  program.  Life  cycle  cost  analysis  is  bet¬ 
ter  for  analyzing  the  differences  among 
alternatives  than  accurately  predicting  future 
costs  Depending  on  the  specific  life  cycle  cost 
model,  the  analysis  output  can  provide  consid¬ 
erable  detail  as  to  where  cost  might  change  as 
the  result  of  program  changes.  The  overall  use¬ 
fulness  of  life  cycle  cost  analysis  is  high  due  to 
the  timely  insight  it  provides  relative  to  a  wide 
range  of  management  decisions. 
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63.2.6  Selection  Criteria  for 

Cost  Risk/WBS  Simulation 
Model 

General:  This  type  of  analysis  model  ag¬ 
gregates  the  cost  uncertainty  due  to  risk  for  any 
number  of  cost  elements  into  a  distribution  of 
the  cost  uncertainty  for  the  entire  project.  Both 
DoD  and  commercial  software  programs  are 
available  to  carry  out  this  type  of  analysis. 

Resource  Requirements:  This  analysis 
method  is  easy  to  use  after  a  few  hours  of 
hands-on  experience.  Available  programs 
come  with  instructions.  However,  obtaining 
and  substantiating  sound  values  for  all  the  cost 
element  uncertainty  information  needed  to  use 
the  method  may  be  difficult.  Ideally,  the  best 
source  of  such  information  would  be  past  expe¬ 
rience  on  similar  programs,  however,  adequate 
information  of  this  type  is  seldom  available. 
The  analysis  computations  can  be  obtained 
within  minutes  after  the  required  data  has  been 
obtained.  Often  the  required  cost  element  un¬ 
certainty  data  must  be  based  on  the  judgment 
of  PMO  personnel.  The  program  managers 
commitment  is  needed  to  assure  PMO  person¬ 
nel  provide  this  information  in  a  timely  man¬ 
ner. 

Application:  The  PMO  survey  indicated 
limited  use  of  this  analysis  technique.  Only 
seven  percent  of  PMO  completing  surveys  re¬ 
ported  specifically  using  the  technique.  Two 
PMOs  reported  it  to  be  useful  for  program 
planning.  There  were  single  reports  of  it  being 
useful  for  POM/BES  preparation,  source  se¬ 
lection,  and  program  status  reporting. 


Output:  The  accuracy  of  the  output  re¬ 
sults  is  limited  by  the  subjective  nature  of  most 
of  the  input  data  used  to  carry  out  this  type  of 
analysis.  The  analysis  does  nothing  to  increase 
visibility  at  a  lower  level  of  detail.  It  does  the 
computation  by  aggregating  detailed  informa¬ 
tion  into  overall  program  cost  risk  information. 
The  overall  usefulness  of  this  type  of  analysis 
for  actually  detecting  risk,  controlling  risk,  or 
reducing  the  impact  of  risk  is  limited.  However, 
it  can  be  used  to  display  cost  risks  known  to  ex¬ 
ist  at  the  cost  element  level  in  an  aggregate 
manner  the  way  some  management  officials 
wish  to  see  it. 

63.2.7  Selection  Criteria  for  Risk 
Factor  Technique 

General:  This  analysis  method  has  been 
widely  used  to  develop  an  estimate  of  the  funds 
required  to  cover  added  costs  resulting  from 
various  risks  felt  to  be  associated  with  each  of 
the  various  work  breakdown  structure  (WBS) 
elements  associated  with  the  program. 

Resource  Requirements:  A  cost  estimate 
broken  out  by  WBS  element  is  a  prerequisite 
for  this  technique.  Obtaining  WBS  element 
risk  factors  from  adequately  qualified  experts 
is  the  bulk  of  the  effort.  The  required  computa¬ 
tions  are  usually  so  easy  that  they  have  been 
carried  out  quickly  with  only  a  hand-held  cal¬ 
culator.  However,  if  a  simple  program  is  not  al¬ 
ready  available  to  carry  out  the  required 
computations,  it  would  be  best  to  quickly  set  up 
the  computations  on  an  electronic  spreadsheet. 
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Application'.  This  analysis  method  can 
only  be  used  when  a  cost  estimate  broken  out 
by  WBS  element  is  already  available.  It  can 
quickly  provide  a  systematically  derived  esti¬ 
mate  of  required  funds  to  cover  the  costs  result¬ 
ing  from  risky  aspects  of  the  program.  It  is 
applicable  to  any  type  of  product  or  size  of  pro¬ 
gram.  It  is  probably  more  applicable  to  smaller 
programs  where  the  resources  and  time  re¬ 
quired  to  apply  more  sophisticated  techniques 
cannot  be  justified.  The  method  can  best  be  ap¬ 
plied  when  PMO  personnel  with  experience  on 
several  other  programs  are  available  to  provide 
judgments  of  the  level  of  risk  involved  with 
each  of  the  WBS  elements. 

Output:  The  output  of  this  analysis 
method  is  an  estimate  of  the  total  funds  re¬ 
quired  to  complete  the  program,  including 
funds  to  resolve  problems  caused  by  risk. 

6.3.2.8  Selection  Criteria 
for  Performance 
Tracking  Technique 

Resource  Requirements :  The  perform¬ 
ance  tracking  technique  requires  a  small 
amount  of  additional  time  above  what  is  nor¬ 
mally  required  to  manage  a  program.  Most  of 
this  time  is  in  the  initial  setup  of  indicators  to 
be  used  and  tracked  in  monitoring  program 
progress.  The  level  of  effort  can  vary  based  on 
contractor  reporting  requirements  as  set  out  in 
the  contract.  With  this  technique,  involvement 
of  the  contractor  is  desirable  in  the  initial  setup 
of  indicators  -  their  level  of  involvement  will 
directly  affect  the  program  office  effort  in  get¬ 
ting  the  indicators  in  place.  The  use  of  a  spread¬ 
sheet  and  PC  are  desirable  but  not  mandatory. 


Once  the  indicators  are  in  place,  minimal  re¬ 
sources  are  required  to  maintain. 

Application:  This  technique  can  be  used 
in  most  of  the  categories  of  Table  6.3-1.  Since 
the  technique  focuses  on  the  monitoring  of 
progress  once  an  item  is  contracted  for,  there  is 
little  value  in  using  it  for  the  Source  Selection 
process. 

Output:  The  output  of  the  technique  is 
very  good  in  general.  If  the  appropriate  indica¬ 
tors  were  selected,  a  quantified  measure  for 
each  potential  problem  area  is  graphically  pre¬ 
sented.  These  are  extremely  useful  for  both 
management  of  the  program  and  communca- 
tion  f'f  the  program  status  to  all  levels  of  deci¬ 
sion-makers. 

633  Technique  Selection 
Summary 

As  discussed,  the  selection  of  a  risk  as¬ 
sessment  and  analysis  technique  is  a  function  of 
the  application,  the  information  needed,  and 
very  importantly,  the  resources  required  versus 
the  resources  available.  Much  of  the  hesitation 
to  implement  many  of  these  techniques  comes 
from  the  concern  that  they  are  too  time  con¬ 
suming,  especially  for  a  small  PMO.  The  results 
of  the  DSMC  survey  and  an  understanding  of 
the  different  techniques  indicate  otherwise. 
The  use  of  Transition  Templates,  Risk  Factor 
Methods,  and  Performance  Tracking  tech¬ 
niques  all  can  be  tailored  to  provide  valuable 
information  without  considerable  expendi¬ 
tures  of  res  urces.  Even  the  network  analysis 
technique  can  be  used  selectively  with  positive 
results  with  less  than  a  major  team  effort.  The 
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key  to  technique  selection  and  successful  risk 
management  implementation  is  the  clear  com¬ 
munication  of  the  objectives,  and  the  integra¬ 
tion  of  risk  management  concepts  and 
activities  into  the  normal  course  of  managing 
the  program. 

6.4  RISK  MANAGEMENT  RESOURCE 

SOURCES 

In  implementing  a  risk  management  ef¬ 
fort,  the  program  manager  must  acquire  and  al¬ 
locate  the  proper  resources  to  tackle  the  job. 
The  program  manager  has  basically  four 
sources  to  task  in  conducting  a  risk  manage¬ 
ment  program. 

•  Program  office 

•  Functional  support  office 

•  Prime  contractors 

•  Support  contractors. 

Each  source  has  been  used  successfully 
by  different  programs  though  most  respon¬ 
dents  to  the  DSMC  Risk  Management  survey 
attempted  to  accomplish  the  majority  of  their 
efforts  within  their  own  PMO. 

Each  source  has  its  strengths  and  weak¬ 
nesses  to  consider  and  certain  criteria  should 
be  used  when  selecting  a  source.  The  first  crite¬ 
ria  is  simply  the  capability  of  the  source  to  ac¬ 
complish  the  risk  management  task.  Capability 
refers  to  the  knowledge  and  understanding  of 
applying  the  risk  techniques.  A  second  criteria 
is  the  availability  of  the  resources  to  accomplish 
the  risk  management  task.  There  are  only  a  fi¬ 


nite  number  of  hours  in  a  day  and  days  in  a 
week.  Though  a  PMO  may  be  highly  skilled 
and  very  experienced,  their  time  commitments 
may  necessitate  finding  other  sources  for  the 
conduct  of  certain  aspects  of  the  risk  manage¬ 
ment  effort.  Objectivity  is  another  criteria 
which  should  be  considered.  By  definition,  the 
PMO  is  an  advocate  of  the  program  and  as 
such,  the  results  of  their  analysis  may  be  viewed 
externally  as  prejudiced.  Depending  on  the 
situation  and  ultimate  use  of  the  results,  this 
may  be  an  important  criteria  for  consideration. 
The  last  factor  considered  is  the  responsiveness 
and  familiarity  of  the  source.  This  criteria  ag¬ 
gregates  the  technical  and  program  knowledge 
of  the  source  with  the  ability  to  respond  to  pro¬ 
gram  changes. 

Table  6.4-1  captures  in  a  matrix  an 
evaluation  of  the  sources  against  these  four  cri¬ 
teria.  In  terms  of  capability,  the  program  of¬ 
fice’s  ability  generally  is  technique  and 
organization  dependent.  Some  program  of¬ 
fices  have  individuals  who  have  developed 
skills  and  are  experienced  in  various  risk  tech¬ 
niques.  Similarly,  those  PMOs  operating  in 
matrixed  organizations  who  turn  to  functional 
support  offices  will  generally  find  capabilities 
that  vary  by  technique  and  organization.  The 
DSMC  Risk  Management  survey  found  an  in¬ 
creasing  awareness  and  capability  among 
PMOs  for  the  conduct  of  risk  management  ef¬ 
forts.  Prime  contractor’s  capabilities  are  also 
dependent  on  the  specific  technique  and  or¬ 
ganization.  In  general,  support  contractors  are 
available  with  the  capability  to  accomplish  risk 
management  tasks.  The  availability  of  the 
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Table  6.4-1  Resource  Selection  Criteria 


CAPABILITY 


AVAILABILITY 


OBJECTIVITY 


RESPONSIVENESS 

FAMILIARITY 


PROGRAM  OFFICE 

FUNCTIONAL  SUPPORT 
OFFICE 

PRIME  CONTRACTOR 

SUPPORT 

CONTRACTOR 

TECHNIQUE  AND 

ORGANIZATION 

DEPENDENT 

TECHNIQUE  AND 

ORGANIZATION 

DEPENDENT 

TECHNIQUE  AND 

ORGANIZATION 

DEPENDENT 

VARIES 

VARIES 

VARIES 

GENERALLY 

AVAILABLE/DIRECT 

COST 

AVAILABLE/DIRECT 

COST 

PROGRAM  AOVOCATE 

STRONGEST 

WEAKEST 

CONTROLLED  BY 
PROGRAM  ADVOCATE 

STRONG 

VARIES 

STRONG 

MODERATE/DIRECT 

COST 

source  obviously  varies  in  the  government.  It  is 
generally  available  from  the  prime  and  support 
contractors,  but  at  a  direct  cost.  In  terms  of  ob¬ 
jectivity,  the  PMO  is  viewed  as  a  program  advo¬ 
cate  and  as  such,  may  be  suspect.  The  support 
contractor  may  be  mete  objective,  but  is  con¬ 
trolled  to  a  degree  by  the  PMO.  The  weakest 
source  in  objectivity  is  the  prime  contractor  for 
obvious  reasons,  and  generally  the  strongest 
source  comes  from  the  independence  of  the 
functional  support  office.  The  PMO  and  the 
prime  contractor  are  rated  as  being  familiar 
with  a  program  and  responsive  to  it.  The  sup¬ 
port  contractor  is  rated  as  moderate  due  to  the 
learning  that  must  be  accomplished  to  conduct 


an  assessment  and  analysis.  Keeping  the  sup¬ 
port  contractor  at  a  certain  level  of  involve¬ 
ment  to  improve  responsiveness  and  familiarity 
is  a  direct  cost  to  the  program.  The  responsive¬ 
ness  and  familiarity  of  the  functional  support 
offices  varies.  They  typically  suffer  to  a  lesser 
degree  from  the  need  to  learn  the  program  in 
order  to  accomplish  their  risk  management 
task. 

In  the  final  assessment,  for  the  selection 
of  a  source  to  accomplish  a  major  part  of  a  pro¬ 
gram’s  risk  management  effort,  the  program 
manager  should  first  look  to  the  PMO  itself. 
The  benefits  in  program  definition  and  under- 


6-19 


standing  in  conducting  the  risk  management 
effort,  in  addition  to  the  information  derived 
for  decision-making  for  the  handling  of  risk, 
merit  the  rigorous  attempt  to  accomplish  the 
effort  in-house.  It  is  appropriate  and  neces¬ 
sary,  however,  to  make  use  of  all  available 
sources  of  expertise  in  the  conduct  of  a  compre¬ 
hensive  risk  management  effort. 

6.5  COMMUNICATING  RISK  DATA 

An  important  aspect  of  risk  manage¬ 
ment  implementation  (which  if  ignored,  can 
make  the  best  risk  assessment  and  analysis  in¬ 
effective  and  incomplete)  is  the  proper  com¬ 
munication  of  risk  data  to  decision-makers. 
The  clear  definition  of  the  terminology  em¬ 
ployed  in  discussing  risk,  the  presentation  of  in¬ 
formation  in  a  clear  and  consistent  format 
within  a  program,  and  the  thorough  documen¬ 
tation  of  the  risk  data  are  the  basics  for  success¬ 
fully  communicating  information  about  risk. 

No  DOD  or  service  standards  exist  for 
the  clear  definition  of  terms  for  risk.  While  this 
handbook  has  presented  a  basic  framework  for 
discussing  risk,  an  argument  can  be  made  that 
no  universal  standard  can  be  employed  to  com¬ 
pare  risk  across  programs.  Given  these  circum¬ 
stances,  the  clear,  unambiguous  definition  of 
the  terminology  used  to  present  risk  related 
data  must  be  accomplished  for  common  under¬ 
standing  among  program  participants  and 
higher  command  levels. 

Beyond  terminology,  for  a  full  under¬ 
standing  of  the  risk  information,  the  process 


and  the  content  of  the  conduct  of  the  risk  man¬ 
agement  effort  must  also  be  captured  and  com¬ 
municated.  The  sources  of  data,  the 
assumptions  made  about  the  program,  the 
methodologies  of  assessment,  analysis,  and 
handling  techniques  used,  and  the  sensitivity  to 
the  risk  data  of  changes  in  the  assumptions, 
must  all  be  consistently  documented  and  com¬ 
municated  to  effectively  implement  risk  man¬ 
agement. 

Last,  though  the  subject  of  risk  is  com¬ 
plex,  the  presentation  of  findings  and  risk  data 
should  be  straightforward  and  in  a  usable  for¬ 
mat.  The  depiction  of  a  cumulative  probability 
distribution  function  from  a  Monte  Carlo  net¬ 
work  analysis  may  be  informative  to  an  ana¬ 
lyst,  but  meaningless  to  the  decision-maker 
reaching  for  a  solution  to  the  problem.  Risk 
data  must  be  presented  in  a  usable  format  that 
communicates  the  essential  elements  of  the 
risk  management  effort. 

6.6  DEVELOPING  A  RISK 

MANAGEMENT  CAPABILITY 

To  successfully  implement  the  risk  man¬ 
agement  process,  the  program  manager  must 
also  address  the  capabilities  of  the  PMO  to  exe¬ 
cute  that  process.  The  discussion  of  risk  man¬ 
agement  implementation  typically  centers 
around  the  evaluation  and  selection  of  the 
tools  and  techniques  to  be  used  and  the  source 
of  manpower  to  use  them.  Much  of  this  chap¬ 
ter  has  focused  on  these  two  topics.  Figure 
6.6-1  illustrates  that  there  are  basically  four 
elements  of  a  risk  management  program  which 
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Figure  6.6-1  Risk  Management  System 


support  the  execution  of  the  risk  management 
process.  Each  of  these  elements  should  be  con¬ 
sidered  when  developing  an  organic  risk  man¬ 
agement  capability. 

While  manpower  and  technique  selec¬ 
tion  are  essential  aspects  of  a  risk  management 
program,  training  and  procedures  are  also 
critical  for  successful  implementation.  Risk 
management,  as  discussed  earlier,  is  a  team  ef¬ 
fort.  Training  in  the  concepts  and  techniques  of 
risk  management  is  required  for  full  under¬ 
standing  and  effective  accomplishment  of  the 
objectives  of  the  risk  management  effort. 
Training  the  PMO  personnel  is  a  necessary  in¬ 


vestment  to  fully  reap  the  benefits  of  the  risk 
management  effort.  Procedures  are  the  docu¬ 
mented  approach  to  executing  the  risk  man¬ 
agement  process.  Whether  contained  in  a 
formal  risk  management  plan  or  not,  proce¬ 
dures  should  be  developed  which  establish  di¬ 
rection  and  responsibility  for  the  conduct  of 
the  risk  management  process. 

6.7  CHAPTER  6  KEY  POINTS 

•  Program  Managers  must 
organize  for  risk  manage¬ 
ment 
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•  Risk  Management  is  a 
team  function 

•  Technique  selection  should 
be  based  on  pre-deter- 
mined  criteria 

•  Proper  communication  of 
risk  information  is  as  im¬ 
portant  as  the  process 

•  Program  Managers  should 
strive  to  develop  their  risk 
management  capability. 
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Chapter  7 

CONTRACTOR  RISK  MANAGEMENT 


7.1  GOVERNMENT 

RESPONSIBILITIES 

In  preparing  a  Request  for  Proposal 
(RFP)  it  is  essential  that  the  procuring  agency 
squarely  face  the  fact  that  risk  management  is 
part  of  the  acquisition  strategy.  A  formal  plan 
of  risk  evaluation  and  reduction  should  be  es¬ 
tablished  by  the  government  very  early  in  each 
acquisition  program.  This  plan  should  be  tai¬ 
lored  to  consider  the  contractor  and  govern¬ 
ment  risks.  The  assessment  and  analysis  of  each 
significant  element  of  program  risk  should  be 
continued  throughout  the  acquisition  cycle. 
The  acquisition  strategy  should  lower  the  risks 
to  reasonably  acceptable  levels.  The  procuring 
agency  should  include  in  the  RFP  requirements 
for  the  offerors  to  describe  their  approach  to 
identifying  and  managing  the  risk  inherent  in 
the  program.  These  would  most  probably  in¬ 
clude  areas  such  as  reliability,  maintainability, 
producibility,  quality,  design,  manufacturing 
technology,  and  research  along  with  many  oth¬ 
ers  too  numerous  to  mention.  In  addition,  the 


RFP  should  include  data  items  such  as  a  Risk 
Management  Plan  and  a  Risk  Assessment  Re¬ 
port  in  order  to  insure  that  the  contractor  will 
seriously  plan  for  risk  management  and  is  con¬ 
tinuously  assessing  the  risk. 

Some  sample  statements  that  could  be 
used  in  an  RFP  include  the  following  (Ref. 
7-1): 

“The  executive  summary  shall  present 
a  proposal  overview,  expected  performance 
including  reliability,  maintainability, 
producibility,  design,  and  supportability  issues, 
work  to  be  accomplished,  trade-offs,  risk  ar¬ 
eas,  schedule,  special  considerations,  and  any 
other  items  necessary  to  briefly  summarize  sali¬ 
ent  proposal  characteristics.” 

Engineering/Design 

‘The  offeror  shall  describe  the  engi- 
neering/technicai  tasks  to  be  accomplished 
during  the  D/V  program  which  contribute  to 
risk  reduction  and  definition  of  the  substanti- 
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ated  system/subsystem  concept.  The  discussion 
shall  contain  the  following  item: 

A  discussion  of  major  technical  risk 
items  associated  with  the  offeror’s  proposed 
system  concept,  including  payoffs  which  will 
potentially  result  from  the  proposed  approach, 
as  well  as  problem  areas.  The  approach  to  de¬ 
termining  the  technical  risks  involved  in  your 
program  and  your  approach  to  reduc’ng  such 
risks  to  acceptable  levels  shall  be  described. 
Key  development  issues  and  the  proposed  solu¬ 
tion  approach  shall  be  identified.  The  discus¬ 
sion  shall  present  the  criteria  to  be  used  to 
evaluate  critical  decision  points  and  informa¬ 
tion  requirement,  and  the  process  to  be  used  to 
develop,  evaluate,  and  implement  fallback 
positions  as  required.” 

R&M 

“Describe  your  approach  to  determin¬ 
ing  the  technical  risk  involved  in  your  R&M 
program  and  your  approach  to  reducing  such 
risks  to  acceptable  levels.  This  discussion  shall 
present  the  criteria  you  plan  to  use  in  determin¬ 
ing  the  criticality  of  technologies,  the  tech¬ 
niques  used  to  evaluate  critical  decision  points 
and  information  requirements,  and  the  process 
used  to  develop,  evaluate,  and  implement  fall¬ 
back  positions  as  required." 

Producibility 

“Describe  the  approach  to  determining 
the  technical  risk  involved  with  the  ensign 
producibility  engineering  program  and  tb-  ap¬ 


proach  to  reducing  such  risks  to  acceptable  lev¬ 
els.  This  discussion  shall  present  the  criteria 
you  plan  to  use  in  determining  the  criticality  of 
technologies,  the  techniques  used  to  evaluate 
critical  decision  points  and  information  re¬ 
quirements,  and  the  process  used  to  develop, 
evaluate,  and  implement  fallback  positions  as 
required.” 

Quality  in  Design 

“Identify  quality  in  design  risks  and  fac¬ 
tor  these  risks  into  design  trade  studies.” 

'  manufacturing  Research/Technology 

Provide  an  assessment  of  the  likeli¬ 
hood  that  the  system  design  concept  can  be 
produced  using  existing  manufacturing  tech¬ 
nology  while  simultaneously  meeting  quality, 
rate,  and  cost  requirements.  Include  in  your 
analysis  and  evaluation  of  the  producibility  of 
the  design  concept:  requirements  for  critical 
process  capabilities  and  special  tooling  devel¬ 
opment,  tests  and  demonstrations  required  for 
new  materials,  alternate  design  approaches, 
anticipated  manufacturing  risk,  potential  cost 
and  schedule  impacts,  and  industrial  base  and 
surge  capabilities.” 

Project  Control  System 

“The  offeror  shall  describe  the  ap¬ 
proach  system  and  methodology  for  risk  man¬ 
agement.  This  discussion  will  include  how 
information  from  the  functional  areas  shall  be 
integrated  into  the  risk  management  process." 
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Manufacturing  Planning 

“Describe  the  initial  manufacturing 
planning  accomplished  in  the  following  areas: 
production  risk,  risk  resolution,  and  identifica¬ 
tion  of  fall  back  positions,  resource  require¬ 
ments,  critical  materials  and  processes,  long 
lead  requirements,  management  systems,  or¬ 
ganization  and  staffing,  and  scheduling.” 

Quality  Assurance 

“Describe  any  QA  risks  you  foresee  for 
this  program  and  actions  planned  to  reduce  the 
risks.” 

Security 

“Operational  Risks 

(a)  Level/Amount  of  Classified:  Iden¬ 
tify  the  levels  of  classification  which  will  be 
processed  as  well  as  the  estimated  hours  per 
month  and  percent  of  total  material  processed 
for  each  category. 

(b)  Sensitivity/Perishability:  Identify 
any  significant  factors  concerning  the  sensitiv¬ 
ity  and/or  perishability  of  the  classified  data. 

(c)  Frequency  of  Processing:  Identify 
the  classified  processing  schedule  which  will  be 
used;  e.g.,  scheduled,  irregular,  sporadic,  ran¬ 
dom.  Assess  the  probability  of  the  exact  hours 
of  classified  use  being  pinpointed  by  unauthor¬ 
ized  personnel.  Describe  any  facts  or  circum¬ 
stances  that  would  make  such  determinations 
difficult.” 


“Technical  Risks 

(a)  Physical  Control  Space  (PCS):  Iden¬ 
tify  the  radius  in  meters  of  the  physical  control 
space  available  around  the  systems/equip¬ 
ment/facility.  Describe  the  barriers,  doors, 
fences,  walls,  etc,  that  define  the  PCS.  Describe 
the  control  exercised  over  the  PCS  during  duty 
and  non-duty  hours.  Describe  other  factors 
which  contribute  to  control,  such  as  visitor  pro¬ 
cedures,  escort  requirements,  searches  of  per¬ 
sonnel  and/or  vehicles,  etc.  (PCS  is  the  area 
within  which  only  personnel  with  Government 
security  clearances  are  allowed  unescorted  ac¬ 
cess.) 

(b)  PCS  Breaches:  Identify  the  type  and 
location  relative  to  the  system  of  any  unfiltered 
telephone  or  communications  lines,  un¬ 
grounded  or  unfiltered  power  lines,  conduits, 
heating  and  air  conditioning  ducts,  water  pipes, 
etc.,  that  transgress  the  established  PCS. 

(c)  Building  Construction:  Describe  the 
building  in  which  the  system  is  housed,  e.g.. 
concrete  block  walls,  aluminum  doors,  no  win¬ 
dows. 

(d)  RED/BLACK  Installation:  Identify 
whether  classified  processors  were  installed  in 
accordance  with  RED/BLACK  criteria  (i.e.. 
installed  in  accordance  with  NACSIM  5203). 

(e)  Shielded  Enclosure:  Identify 
whether  classified  processors  are  operated 
within  an  RF  shielded  enclosure.” 
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Evaluation  Summary 

‘The  overall  evaluation  of  each  pro¬ 
posal  may  include  on-site  inspections  and  re¬ 
sults  of  pre-award  surveys  to  provide 
information  to  the  Source  Selection  Authority 
regarding  offerors  current  and  future  capabil¬ 
ity  to  perform  all  aspects  of  the  program.  Risk 
assessment  associated  with  the  major  areas  of 
the  program  will  be  accomplished.  In  assessing 
risk,  an  independent  judgment  of  the  probabil¬ 
ity  of  success,  the  impact  of  failure,  and  the  al¬ 
ternatives  available  to  meet  the  requirements 
will  be  considered.” 

7.2  CONTRACTOR  RESPONSIBILITY 

The  contractor  should  be  made  aware 
through  written  language  in  the  contract  that 
the  information  contained  in  the  DIDs  will  be 
used  for  risk  analysis.  It  should  be  the  contrac¬ 
tors  responsibility  to  make  a  thorough  assess¬ 
ment  of  risks  in  proposing  the  contractual 
effort.  Sufficient  information  should  be  in¬ 
cluded  in  the  proposal  to  convince  the  govern¬ 
ment  that  the  contractor  has  recognized  and 
quantified  the  risk  inherent  in  the  program. 
The  proposal  should  identify  areas  where  ac¬ 
tions  by  the  government  can  aid  in  risk  reduc¬ 
tion.  These  can  include  items  such  as  long  lead 
funding  and  the  necessity  for  approval  of  prior¬ 
ity  status  for  materials 

In  proposing  a  risk  management  sys¬ 
tem,  the  contractor  should  highlight  how  he 
can  use  existing  internal  systems  (e.g.,  his  cost 
and  schedule  control  system)  to  provide  infor¬ 


mation  on  risk.  The  contractor  should  also  con¬ 
centrate  on  how  he  can  include  risk 
considerations  in  normal  management  prac¬ 
tices  and  in  the  various  items  of  data  provided 
to  the  government. 

73  CHAPTER  7  KEY  POINTS 


•  RFPs  should  request  infor¬ 
mation  about  the  process 
that  the  contractor  will  use 
to  manage  risk 

•  Contracts  should  include 
deliverables  containing  in¬ 
formation  regarding  risk 

•  Contractors  should  formal¬ 
ize  risk  management. 


Reference 


7-1  Extracts  from  ATF  RFP  Attachment  X, 
F33657-85-R-0062. 
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Chapter  8 

THE  FUTURE  OF  RISK  MANAGEMENT 


Risk  is  a  fascinating  subject.  To  those 
who  try  to  understand,  it  looks  a  bit  different 
each  day  -  like  a  crystal  reflecting  light  differ¬ 
ently  depending  on  the  angle  of  view.  Studying 
risk  leads  a  person  through  a  wide  range  of  aca¬ 
demic  disciplines  from  rigid  mathematical 
probabilities  through  sophisticated  computer 
models  into  the  behavioral  sciences  and  on  to 
the  psychology  of  risk  takers  and  risk  avoiders. 

There  are  many  students  and  practitio¬ 
ners  who  are  convinced  they  have  the  single 
right  answer  to  the  understanding  and  manage¬ 
ment  of  risk.  Many  confuse  the  tool  with  the  re¬ 
sult.  Academics  want  to  quantify  and  analyze. 
Bureaucrats  seek  more  information,  avoid 
commitment,  and  criticize.  Program  managers 
live  with  risk.  They  “own”  the  problem. 

Risk  management,  in  the  context  of  in¬ 
terest  here,  is  being  practiced  within  DoD.  Not 
as  much  as  it  should  be,  but  more  than  the  crit¬ 
ics  would  allow.  Program  managers  deal  with 
risk  daily.  Frequently,  they  think  of  it  as  man¬ 
agement  without  suspecting  it  is  risk  manage¬ 


ment.  To  that  extent,  the  criticism  is 
unjustified.  On  the  other  hand,  many  managers 
do  not  seek  to  identify  and  resolve  risks  early, 
but  rather  deal  only  with  those  risks  that  ap¬ 
pear  today.  Finally,  there  are  those  who  at¬ 
tempt  to  obscure  risk  in  the  practice  of  “not  on 
my  watch”  program  management.  To  that  ex¬ 
tent  the  criticism  is  very  justified. 

Risk  management  can  be  practiced  bet¬ 
ter  than  it  is.  Tools  are  available.  They  are  not 
perfect,  but  they  can  be  improved.  We  are  not 
living  up  to  the  existing  capacity.  The  body  of 
knowledge,  available  tools,  and  the  computer 
power  is  available  to  make  a  major  step  for¬ 
ward  in  risk  management. 

If  risk  is  to  be  properly  managed,  it 
must  be  recognized,  acknowledged,  and  ac¬ 
cepted.  It  must  be  taken  out  of  the  closet.  A 
fundamental  culture  change  is  necessary  with 
regard  to  risk. 

•  Program  Managers  must  be 
penalized  for  not  communi- 
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eating  risk  rather  than  for 
doing  so. 

•  DoD  “corporate  manage¬ 
ment”  must  insist  on  seeing 
and  hearing  about  risk  ana 
then  have  the  courage  to 
support  the  risk  takers. 

Risk  management  currently  suffers 
from  a  limited  vocabulary  and  a  lack  of  stan¬ 
dard  definitions.  Communication  on  the  sub¬ 
ject  would  be  aided  immensely  by  treatment  of 
the  vocabulary  and  definitions  problem. 

Risk  management  currently  lacks  stan¬ 
dardized  procedures  and  techniques.  At  this 
stage  of  development,  lack  of  standardization 
is  NOT  a  fault.  There  are  many  very  intelligent 
people  who  will  devise  ingenious  and  effective 
techniques  if  given  the  requirements  and  the 
freedom  to  be  creative. 

In  the  spectrum  of  the  risk  management 
process,  the  weakest  area  at  present  is  that  of 
“quantifying  expert  judgment”.  Risk  identifica¬ 
tion  focuses  on  capturing  the  knowledge  and 
judgment  of  experts.  Risk  assessment  and 
analysis  deals  largely  with  mathematical  state¬ 
ments  and  quantified  results.  Transitioning 
from  the  English  language  statements  of  ex¬ 
perts  to  the  mathematical  expressions  required 
by  the  analytical  tools  is  done  inconsistently. 

Strengthening  this  area  will  significantly 
improve  the  risk  management  process.  Some 
rudimentary  surveys  of  on-going  research  in 
the  field  indicates  that  this  pre »  lem  is  a  natural 
for  the  application  of  expert  systems  technol¬ 
ogy  and  is  in  fact  being  worked. 


Risk  handling  is  that  portion  of  the 
process  where  the  program  manager  attempts 
to  reduce  or  contain  the  risks  that  have  been 
identified,  quantified,  and  analyzed.  Risk  han¬ 
dling  is  discussed  in  Section  4.5  and  5.14  which 
cover  only  a  small  amount  of  this  document. 
That  illustrates  the  disproportionate  share  of 
thought  and  literature  that  has  gone  to  risk  as¬ 
sessment  and  risk  analysis  at  the  expense  of  risk 
handling.  The  program  manager  is  ultimately 
left  with  the  question  “What  can  I  do  about  it?" 
Risk  management  should  benefit  greatly  from 
future  efforts  concentrated  in  developing  and 
documenting  new  ideas  in  risk  handling  tech¬ 
niques. 
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APPENDIX  A 

RISK  SOURCES  -  AN 
ABBREVIATED  LIST 


A-l 


1.  TECHNICAL  RISK  SOURCES 


1.1  MAJOR  STATE-OF-THE- 

ART  ADVANCE 

These  are  problems  that  could  cause  de¬ 
viations  from  the  planned  program  resulting 
from  greater  than  anticipated  state-of-the-art 
advances.  This  includes  areas  such  as: 

•  Complexity/difficulty  in 
meeting  requirements 

•  Percent  proven  technology 

•  Experience  in  the  field 
needed 

•  Lack  of  work  on  similar 
programs 

•  Special  resources  needed 

•  Operating  environment 

•  Required  theoretical  analy¬ 
sis 

•  Degree  of  difference  from 
existing  technology. 


1.2  NUMEROUS  STATE-OF-THE- 
ART  ADVANCES 

Deviations  from  the  planned  program 
could  result  from  a  greater  number  of  areas 
than  anticipated  requiring  advanced  state-of- 
the-art  techniques  and  development. 


U  STATE-OF-THE-ART  ADVANCE 
PROGRESS 

Slower  than  expected  progress  in  ad¬ 
vancing  the  state-of-the-art  could  affect  the 
planned  program. 


1.4  LACK  OF  SUPPORTING 
STATE-OF-  THE-ART  ADVANCES 

State-of-the-art  advances  expected 
from  other  programs  may  not  be  as  expected 
and  can  have  a  significant  affect  on  the  present 
program. 

1.5  FIELD  FAILURES  OF  STATE-OF- 
THE-ART  ADVANCES 

Field  failures  of  state-of-the-art  equip¬ 
ment  types  that  were  assumed  to  be  ready  for 
incorporation  into  the  planned  program  can 
have  a  negative  effect  on  the  program. 

1.6  OPERATING  ENVIRONMENT 

The  new  system  may  be  required  to  per¬ 
form  in  an  unusually  harsh  environment  which 
would  cause  problems  with  the  program. 

1.7  UNIQUE  HARSH  REQUIREMENTS 

Significant  differences  between  exist¬ 
ing  design  technology  and  that  required  for 
success  of  the  new  system  can  cause  deviations 
in  the  plans  for  the  new  system. 

1.8  PHYSICAL  PROPERTIES 

If  the  dynamics,  stress,  thermal,  or  vi¬ 
bration  physical  property  requirements  are  dif¬ 
ferent  than  originally  expected,  the  planned 
program  may  not  achieve  its  original  goals. 

1.9  MATERIAL  PROPERTIES 

Material  property  requirements  beyond 
what  is  usually  expected  could  influence  the 
planned  program. 
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1.15  INTEGRATION/INTERFACE 


1.10  RADIATION  PROPERTIES 

Increased  radiation  stress  resistance  re¬ 
quirements  can  result  in  changes  to  the  pro¬ 
gram  from  the  original  plan. 


1.11  MODELING  VALIDITY 

Models  used  in  developing  mathemati¬ 
cal  and  physical  predictions  can  contain  inaccu¬ 
racies  affecting  the  program. 


1.12  TESTING  INCONSISTENCIES 

Inconsistent  field  test  results  can  cause 
increased  technical  risk  and  require  retesting. 


1.13  TEST  FACILITY  COMPATIBILITY 

Suitable  test  facilities  may  not  be  avail¬ 
able  during  the  required  time  frame  and  cause 
significant  scheduling  problems. 


1.14  EXTRAPOLATION 
REQUIREMENTS 

During  the  conduct  of  the  program,  the 
need  for  extensive  extrapolation  using  field  test 
results  may  hamper  the  assessment  of  the  pro¬ 
gram  under  actual  deployment  conditions. 


New  and  unique  design  adaptability, 
compatibility,  interface  standard,  and  inter¬ 
operability,  etc.,  requirements  can  create  situ¬ 
ations  that  are  not  compatible  with  the  original 
planned  program. 

1.16  SURVIVABILITY 

New  requirements  for  nuclear  harden¬ 
ing,  chemical  survivability,  etc.,  may  require  re¬ 
vised  planning  in  order  to  meet  original  or  new 
goals. 

1.17  SOFTWARE  DESIGN 

Unique  software  test  requirements  and 
unsatisfactory  software  test  results  could  result 
in  the  generation  of  variances  to  the  basic 
planned  program. 

1.18  SOFTWARE  LANGUAGE 

A  new  computer  language  or  one  unfa¬ 
miliar  to  most  of  those  responsible  for  planning 
and  writing  computer  software  could  change 
the  entire  perspective  of  the  planned  program. 

1.19  RELIABILITY 

Failure  to  properly  forecast  system  reli¬ 
ability  or  failure  to  obtain  predicted  reliability 
growth  could  cause  the  program  to  deviate 
from  its  desired  course. 
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1.20  MAINTAINABILITY 

Failure  to  obtain  d*.  sired  maintenance 
performance  with  a  design  that  is  compatible 
with  proven  maintainability  procedures  can  re¬ 
quire  changes  in  the  maintenance  concept. 

1.21  FAULT  DETECTION 

Fault  detection  techniques  may  reveal  a 
failure  to  obtain  designed  performance  and  re¬ 
quire  modification  to  the  program. 

2.  PROGRAMMATIC  RISK 
SOURCES 


2.1  HIGHER  AUTHORITY  ACTION 

RISK  CATEGORY 

2.1.1  Category  Definition 

Risks  falling  within  this  category  result 
from  decisions  of  actions  by  higher  levels  of 
authority  -  generally  by  people  knowing  its  im¬ 
pact  on  the  program  but  who  are  addressing 
larger  issues. 

2.1.2  Specific  Higher  Authority 
Action  Risks 

Priority  Risk  -  Problems  that  could  af¬ 
fect  the  planned  program  resulting  from  chang¬ 
ing  priority  assigned  to  the  program  and 
thereby  timely  access  to  testing  facilities,  funds, 
materials,  etc. 

Decision  Delay  Risks  -  Disruption  of  the 
planned  program  schedule  resulting  from  de¬ 
lays  in  obtaining  higher  level  approval  to  award 


contracts,  proceed  to  the  next  phase,  etc.,  can 
cause  program  problems. 

Inadequate  SPO  Authority  Risks  - 
Planned  program  delays  resulting  from  the 
SPO  not  being  given  adequate  authority  to 
manage  the  program  including  having  the 
authority  to  make  timely  cost,  schedule,  and 
performance  trade-off  decisions  can  be  a  sig¬ 
nificant  risk. 

Joint  Service  Program  Decision  Risks  - 
Problems  and  delays  that  could  disrupt  the 
planned  program  resulting  from  reduced  joint 
service  participation  or  other  user  decisions. 

Service  Roles  and  Mission  Changes  - 
Problems  that  will  cause  deviations  from  the 
planned  program  resulting  from  changing  serv¬ 
ice  roles  and  missions  which  significantly  alter 
the  planned  use  of  the  system. 

Concurrency  -  Concurrent  develop¬ 
ment  or  the  preparation  for  production  can 
cause  deviations  from  the  planned  program. 
Concurrency  often  results  in  discovery  of  prob¬ 
lems  at  a  time  when  a  cost  premium  must  be 
paid  to  resolve  problems  and  keep  the  program 
on  or  near  the  original  schedule. 

Funding  Constraints  -  Lack  of  timely  re¬ 
ceipt  of  programmed  funds  as  anticipated  can 
cause  deviation  from  the  original  plan. 

Program  Stretch  Out  -  Direction  to  slip 
the  program  schedule  from  the  original  plan 
will  cause  funding  problems. 

Continuing  Resolution  -  The  require¬ 
ment  to  execute  a  program  for  a  period  of  time 
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with  funds  provided  by  a  continuing  resolution 
and  the  resulting  constraints  associated  with 
the  continuing  resolution  create  unforeseen 
problems. 

National  Objectives  and  Strategies  - 
Changes  in  national  objectives  and  strategies 
will  cause  deviations  to  the  planned  program. 


2.2  NON-PROGRAM  EVENT  OR 

ACTION  CATEGORY 

2.2.1  Category  Definition 

Risks  falling  within  this  category  result 
from  varied  events,  policy  changes,  decisions, 
or  actions,  not  aimed  specifically  at  the  pro¬ 
gram,  but  disrupting  original  plans  in  some 
manner. 

2.2.2  Specific  Non  Program 
Event  or  Action  Risks 

Inflation  -  Significantly  higher  levels  of 
inflation  than  originally  forecast  can  create 
funding  problems. 

Legislation  -  Higher  taxes,  new  labor 
laws  affecting  pay  and  benefits,  social  security 
increases,  etc.,  can  cause  significant  funding 
problems. 

Environmental  Impact  -  Natural  disas¬ 
ters  such  as  fires,  floods,  storms,  earthquakes, 
etc.,  can  cause  major  schedule  delays  and  cost 
problems. 

Source  Selection  Protests  -  Source  selec¬ 
tion  award  protests  and  related  legal  actions 


can  delay  the  start  of  a  program  with  resulting 
schedule  and  cost  problems. 

Labor  Disputes  -  Labor  difficulties  such 
as  strikes,  lock  outs,  slowdowns,  etc.  will  affect 
work  on  the  program. 

Threat  Changes  -  Threat  changes  re¬ 
quiring  changes  in  schedule  and  performance 
objectives  will  cause  deviations  in  schedule  and 

Operating  Policies  -  Changes  in  operat¬ 
ing  policies  impacting  system  or  system  support 
requirements  can  cause  tl  e  program  to  vary 
from  the  original  plan. 

New  Regulations  -  Added  workload  or 
time  requirements  brought  about  by  new  Con¬ 
gressional,  DoD,  or  service  direction  or  policy 
can  create  significant  variances  to  the  basic 
planned  program. 

23  PRODUCTION  PROBLEM 

RISK  CATEGORY 

23.1  Category  Definition 

Risks  falling  within  this  category  result 
from  unanticipated  problems  associated  with 
the  process  of,  or  resources  needed  for  system 
production. 

23.2  Specific  Production 
Problem  Risks 

Design  Stability  -  The  lack  of  design  sta¬ 
bility  during  the  production  phase  can  create 
serious  problems  in  meeting  production  sched¬ 
ules  and  cost  goals. 
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Familiarization  -  If  contractor  person¬ 
nel  are  not  familiar  with,  and  do  not  have  expe¬ 
rience  producing  similar  systems  or  equipment, 
problems  in  executing  the  planned  program 
can  occur. 

Scarce  Resource  -  Shortages  of  critical 
materials,  components,  or  parts  can  delay  pro¬ 
duction  and  ultimately  increase  costs. 

Tolerance  Levels  -  Closer  than  usual 
tolerance  levels  and  difficulties  in  achieving 
these  tolerance  requirements  are  a  subset  of  fa¬ 
miliarization  and  can  cause  program  problems. 

Vendor  Base  -  A  shortage  of  an  ade¬ 
quate  number  of  qualified  vendors  necessaiy  to 
ensure  adequate  price  competition  and  a  satis¬ 
factory  supply  quantity  base  can  cause  both 
schedule  and  cost  problems. 

Capacity  -  The  lack  of  facilities  and 
tools  to  produce  at  the  desired  rate  (rate  tool¬ 
ing)  can  prevent  the  production  flow  from 
reaching  the  desired  level. 

Excessive  Lead  Times  -  If  longer  than 
expected  lead  times  for  critical  components  or 
services  are  experienced  then  the  program  will 
slip. 

Advance  Buy  Authorization  Limitations 
-  Long  lead  time  requirements  can  create 
problems  if  there  is  not  sufficient  advanced  buy 
funding  to  meet  the  needs  of  the  program. 

Production  Readiness  -  If  the  contrac¬ 
tor  fails  to  be  adequately  prepared  for  produc¬ 
tion,  slippage  will  occur  in  the  program. 


2.4  IMPERFECT  CAPABILITY 

RISK  CATEGORY 

2.4.1  Category  Definition 

Risks  falling  within  this  category  are  the 
result  of  people,  organizations,  or  facilities  not 
performing  as  well  as  desired  or  expected. 

2.4.2  Specific  Imperfect 
Capability  Risks 

Underbidding  -If  the  contractors  under¬ 
bids  or  buys-in  to  get  contracts  and  fails  to  pro¬ 
vide  the  desired  products  and  services  on 
schedule  and  within  budget,  then  the  planned 
program  will  be  significantly  affected. 

Subcontractor  Control  -  If  the  prime 
contractor  does  not  maintain  adequate  control 
of  subcontractor  quantity,  schedule,  and  cost 
performance,  then  the  planned  program  will 
not  make  its  original  goals. 

Lack  of  Financial  Strength  -  If  one  or 
more  contractors  has  not  been  able  to  ade¬ 
quately  finance  program  requirements,  the  re¬ 
quired  work  will  be  delayed  or  curtailed. 

Communication  -  Problems  that  could 
cause  deviations  from  the  planned  program 
can  result  from  failure  of  the  subcontractor’s 
and  contractor’s  personnel  to  keep  prime  con¬ 
tractor  and  SPO  management  informed  of 
problems  and  potential  problems  in  a  timely 
manner.  Likewise,  communication  problems 
can  occur  if  management  fails  to  fully  commu¬ 
nicate  direction  to  all  involved  in  the  program 
in  a  timely  manner. 

Forced  Placement  -  If  the  program  is 
saddled  with  second  string  personnel  and  man- 
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agers  either  in  the  SPO  or  at  key  contractors, 
then  serious  counterproductive  events  could 
occur  causing  program  perturbations. 

2.5  OTHER  PROGRAM  PROBLEMS 

RISK  CATEGORY 

2.5.1  Category  Definition 

Risks  falling  within  this  category  are 
generally  somewhat  different  from  program  to 
program  due  to  the  unique  nature  or  require¬ 
ments  of  the  product  and  program.  This  cate¬ 
gory  does  not  include  production  related  risks 
which  have  been  placed  in  a  separate  category 
but  could  be  considered  a  subset  of  this  cate¬ 
gory. 

2.5.2  Specific  Other  Program 
Problem  Risks 

Available  Skills  -  The  shortage  of  avail¬ 
able  personnel  with  the  needed  technical,  man¬ 
agement,  and  other  skills  to  carry  out  PMO 
and  contractor  activities  could  create  problems 
that  affect  the  planned  program. 

Security  Clearances  -  Any  delays  in  ob¬ 
taining  required  personnel  security  clearances 
and  facility  clearances  will  have  a  significant 
impact  upon  the  basic  program. 

Secure  Test  Requirements  -  The  testing 
of  classified  equipment  can  cause  difficulties 
that  are  associated  with  testing  classified 
equipment. 

Test  Safety  -  Problems  that  could  cause 
deviations  from  the  planned  program  can  re¬ 


sult  from  the  new  or  unique  requirements  that 
testing  be  non-destructive  or  that  it  not  inter¬ 
fere  with  other  activities. 

Weather  -  Unusually  severe  weather  re¬ 
lated  test  program  delays  can  cause  slippage 
and  cost  overruns  to  the  planned  program. 

Site  Survey  Results  -  Historical  or  ar¬ 
chaeological  site  survey  findings  could  delay 
site  construction  and  cause  significant  deploy¬ 
ment  problems. 

Common  Support  Equipment  -  If  com¬ 
mon  support  equipment  is  not  available  as  re¬ 
quired  to  operate  ;md  maintain  the  system, 
then  the  planned  program  will  suffer  schedule 
and  cost  problems. 

3.  SUPPORTABILITY  RISK 

SOURCES  (Ref.  1) 


3.1  DELAYED  DEFINITION  OF 

LOGISTICS  CRITERIA 

Delayed  decisions  on  reliability  and 
supportability  requirements  result  in  subop¬ 
timum  support.  Once  the  design  is  committed, 
the  options  become  limited.  Many  early  fighter 
aircraft  suffered  from  having  design  optimized 
for  performance  without  comparable  attention 
to  support  aspects  such  as  maintenance  accessi¬ 
bility  and  spare  parts  reliability.  As  a  result, 
turn  around  times  and  operation  and  support 
(O&S)  costs  were  excessive  and  manpower  re¬ 
quirements  for  some  aircraft  models  ap¬ 
proached  100  maintenance  man-hours  per 
flight  hour  (MMH/FH). 
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3.2  IMPACT  OF  ENGINEERING 
CHANGES 

A  high  number  of  design  changes  made 
during  the  development  program  can  over¬ 
whelm  Integrated  Logistics  Support  (ILS) 
planning  and  create  an  inability  to  fully  reflect 
ILS  and  O&S  cost  considerations  in  engineer¬ 
ing  change  decisions. 


33  LATE  ESTABLISHMENT  OF 

READINESS  AND 

SUPPORTABILITY  OBJECTIVES 

The  system  engineering  process  is  a  key 
factor  in  identifying  and  attaining  realistic 
readiness  and  supportability  objectives.  If  a 
well  organized  process  is  not  started  at  the 
program  inception  and  continued  throughout 
the  development  phases,  then  the  program 
risks  are: 

•  Increased  design,  develop¬ 
ment,  and  O&S  costs 

•  Schedule  delays 

•  Degraded  readiness 
factors. 


3.4  UNREALISTIC  R&M 

REQUIREMENTS 

The  establishment  of  unrealistic  Reli¬ 
ability  and  Maintainability  (R&M)  require¬ 
ments  (as  part  of  the  Pre-Program  Initiation  of 
Concept  Exploration  (CE)  phases)  can  lead  to 
increased  design  and  development  costs  in¬ 
curred  as  a  result  of  excessive  design  iterations. 
This  in  turn  can  cause  program  delays  and 


costly  program  support  system  restructuring  in 
later  phases. 

3.5  ACQUISITION  STREAMLINING 

The  new  DoD  initiative  on  acquisition 
streamlining  may  impose  restrictions  on  the 
ILS  Manager  as  well  as  the  designer  early  on  in 
the  definition  of  requirements.  Although  in¬ 
tended  to  decrease  cost  and  improve  efficiency, 
casual  application  of  such  guidance  could  re¬ 
sult  in  a  loss  of  standardization,  attendant  cost 
increases,  and  the  loss  of  documented  lessons 
learned  experience. 

3.6  FAILURE  TO  APPLY  LSA  DURING 

CONCEPT  EXPLORATION 

Failure  to  participate  in  the  definition 
of  system  concepts  can  produce  a  system  design 
in  follow-on  phases  that  does  not  meet  suppor¬ 
tability  objectives  and  requires  excessive  or  un¬ 
attainable  operation  and  support  (O&S)  costs 
as  well  as  manpower  to  meet  the  readiness  ob¬ 
jectives. 

3.7  INVALID  APPLICATION  OF 

COMPONENT  R&M  DATA 

Design  and  manufacture  determines 
the  mean  life  and  failure  rate  of  components 
when  viewed  in  isolation.  When  the  parent  ma¬ 
terial  system  is  engaged  in  its  military  opera¬ 
tional  role,  these  same  components  should  be 
expected  to  exhibit  replacement  rates  substan¬ 
tially  higher  than  their  handbook  value  or  in- 
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herent  reliability  alone  would  indicate.  The 
consequences  of  improperly  computed  mate¬ 
rial  replacement  rates  are  invalid  manpower 
requirements,  incorrect  supply  support 
stockage  lists,  and  invalid  repair  level  analyses. 


3.8  FAILURE  TO  STRUCTURE/TAILOR 

LSA  REQUIREMENTS 

Failure  to  establish  a  Logistics  Support 
Analysis  (LSA)  plan  that  is  specifically  de¬ 
signed  to  meet  the  needs  of  the  material  system 
can  result  in:  excessive  costs;  the  performance 
of  unwanted  analysis  while  failing  to  complete 
needed  studies;  and  the  development  of  exces¬ 
sive  documentation  while  overlooking  critical 
information  needs.  ILS  lessons  learned  reports 
and  discussions  with  ILS  Managers  have  pro¬ 
vided  numerous  examples  of  these  deficien¬ 
cies. 


3.9  LACK  OF  LCC  IMPACT  ON 
DESIGN  AND  LOGISTICS 
SUPPORT  PROCESS 

Life  Cycle  Cost  (LCC)  is  most  effective 
when  it  is  integrated  into  the  engineering  and 
management  process  that  makes  design  and  lo¬ 
gistics  engineering  choices.  This  integration 
must  start  with  program  .  itiation.  Once  the 
ability  to  influence  design  is  lost,  it  is  very  diffi¬ 
cult  and  always  more  costly  to  re-establish. 
Most  performance  and  schedule  risks  have  cost 
impacts.  Performance  risks  result  from  re¬ 
quirements  which  are  very  costly,  or  from  engi¬ 
neering  requirements  beyond  foreseeable 


technical  capabilities  for  hardware  develop¬ 
ment.  The  result  can  be  increased  cost  from  de¬ 
sign,  development,  and  test  of  a  replacement 
item;  contract  termination  costs;  increased 
program  buy,  and  increased  O&S  costs.  Sched¬ 
ule  changes  can  increase  costs  whether  they  are 
shortened  or  lengthened. 

3.10  ACCELERATED  PROGRAMS 

Anaccelerated  system  development 
program  may  be  required  to  overcome  a  criti¬ 
cal  deficiency  in  an  existing  military  capability. 
This  “streamlining”  can  pose  the  risk  of  delay¬ 
ing  design  maturation  with  frequent  configura¬ 
tion  changes  occurring  in  late  development 
possibly  continuing  during  initial  production 
and  deployment.  The  added  time  required  to 
modify  LSA  records  (LSAR)  and  update  ILS 
elements  can  lead  to  an  initial  period  of  de¬ 
creased  system  readiness. 

3.11  IMPROPER  CONTRACTING 

FOR  SUPPORT 

The  major  risk  area  in  ILS  contracting, 
in  terms  of  impact  and  the  probability  of  its  oc¬ 
currence,  is  the  failure  to  properly  contract  for 
data,  materials,  and  services.  Included  are  fail¬ 
ures  involving  contractual  promises  by  the 
Government  to  furnish  material  and  services 
and  the  imposition  of  unrealistic  delivery  of 
performance  schedules.  Impacts  may  include 
degraded  support  and  readiness,  cost  growth, 
and  when  repeatedly  exposed,  the  loss  of  tax¬ 
payers’  goodwill  and  confidence. 
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3.12  DELAYED  OR  INADEQUATE 
LOGISTICS  TEST  AND 
EVALUATION  (T&E)  PLANNING 


The  main  thrust  of  the  formal  Develop¬ 
ment  Test  and  Evaluation  (DT&E)  and  Opera¬ 
tional  Test  and  Evaluation  (OT&E)  programs 
is  to  evaluate  system  level  performance.  Logis¬ 
tics  test  and  evaluation  has  an  additional  focus 
on  component  evaluation  and  on  the  adequacy 
of  the  ILS  elements  that  comprise  the  logistic 
support  structure.  Failure  by  the  ILS  Manager 
to  participate  effectively  in  the  initial  develop¬ 
ment  of  the  Test  and  Evaluation  Master  Plan 
(TEMP)  during  the  CE  Phase  risks  the  exclu¬ 
sion  of  critical  logistics  T&E  and  the  omission 
of  the  ILS  test  funds  required  in  the  program 
and  budget  documents. 


3.13  INADEQUATE  PLANNING  FOR 
DATA  UTILIZATION 


Collecting  data  without  detailed  plan¬ 
ning  for  its  use  can  lead  to: 


•  A  mismatch  of  data  collec¬ 
tion  information  require¬ 
ments 

•  Failure  to  accomplish  the 
intended  purpose  of  the  as¬ 
sessment  (such  as  the  up¬ 
date  of  supply  support  and 
manpower  requirements 
and  the  identification  and 
correction  of  design  defi¬ 
ciencies). 


3.14  INCOMPLETE  OR  DELAYED 

SUPPORT  PACKAGE 

Without  an  adequate  test  support  pack¬ 
age  on  site  ready  to  support  the  scheduled  test, 
it  may  be  possible  to  start  testing,  but  the 
chances  are  low  of  continuing  on  schedule.  A 
support  system  failure  could  cause  excessive 
delays,  which  can  incur  a  schedule  slippage  and 
increased  test  cost  due  to  on-site  support  per¬ 
sonnel  being  unemployed  or  for  the  cost  of  fa¬ 
cilities  which  are  not  being  properly  used. 


3.15  INCOMPLETE  OR 

INACCESSIBLE  DATA 

Without  sufficient  data  being  available 
from  each  test,  and  used  properly  for  planning 
subsequent  tests,  it  is  not  possible  to  evaluate 
the  adequacy  of  the  system  to  meet  all  of  its 
readiness  requirements.  Without  accurate  fail¬ 
ure  rates,  system  and  component  reliability 
cannot  be  determined.  Without  cause  of  failure 
established,  Failure  Modes  Effects  and 
Criticality  Analysis  and  Repair  of  Repairables 
Analysis  cannot  be  accomplished.  Integral  to  a 
data  management  system  is  the  retrieval  and 
reduction  of  data  as  well  as  the  collection  and 
storage.  Essential  to  any  test  program  is  the 
ability  to  document  and  collect  results  so  that 
they  are  readily  available  to  both  the  engineer 
and  logistician  for  analysis  at  completion  of  the 
test  program.  Lacking  the  necessary  data,  sys¬ 
tem  design  and  ILS  progress  cannot  be  estab¬ 
lished,  problems  cannot  be  identified,  and  ad¬ 
ditional  testing  may  be  required. 


A-10 


3.16  UNREALISTIC  SCENARIOS 


A  subtle  risk,  particularly  during  devel¬ 
opment  testing,  and  one  which  can  have  lasting 
impact  on  the  viability  of  a  program,  is  testing 
to  an  unrealistic  scenario.  Realism  does  not 
necessarily  mean  that  the  stresses  put  on  the 
system  under  test  must  duplicate  those  of  ac¬ 
tual  service,  since  in  most  cases  this  is  impracti¬ 
cal;  it  does  mean,  however,  that  the  test  is 
planned  to  simulate  the  conditions  as  closely  as 
possible  and  differences  are  carefully  docu¬ 
mented.  Perhaps  more  significant  in  ILS  test¬ 
ing  than  stresses  applied,  is  the  quality  and  skill 
level  of  personnel  maintaining  and  operating 
equipment.  It  is  expected  during  development 
testing,  that  highly  skilled  personnel  will  be  op¬ 
erating  and  maintaining  the  equipment,  since 
the  main  purpose  of  development  testing  is  to 
evaluate  the  hardware  itself  and  to  see  if  it 
demonstrates  the  required  performance.  Dur¬ 
ing  operational  testing,  however,  the  purpose 
of  the  test  is  to  see  how  the  system  operates  un¬ 
der  actual  conditions.  Moreover,  useful  data 
can  only  be  obtained  if  it  is  maintained  and  op¬ 
erated  by  personnel  having  the  same  skill  levels 
and  training  as  the  personnel  planned  to  oper¬ 
ate  and  maintain  the  system  when  deployed  in 
the  field.  If  operational  testing  is  staffed  with 
military  personnel  having  much  more  experi¬ 
ence  and  skill  than  can  be  expected  when  de¬ 
ployed,  the  operational  testing  will  give  an  un¬ 
realistically  favorable  evaluation,  which 
though  favorable  to  the  system,  provides  mis¬ 
leading  information  resulting  in  invalid  conclu¬ 
sions. 


3.17  ACCELERATED  PROGRAMS 

Compressed  schedules  increase  the  de¬ 
mand  for  critical  assets  during  the  time  of  nor¬ 
mal  asset  shortages  which  can  create  un¬ 
recoverable  delays. 

3.18  SCHEDULE  SLIPPAGE 

Failure  to  understand  how  a  schedule 
slippage  in  one  functional  element  impacts  the 
other  elements  and  milestone  events  can  ulti¬ 
mately  delay  the  entire  program. 

3.19  DELAYED  FACILITIES  PLANNING 

Failure  to  perform  timely  facility  plan¬ 
ning  can  result  in  substantial  deployment  de¬ 
lays. 

3.20  UPDATING  THE  DEPLOYMENT 
PLAN 

Failure  to  keep  the  deployment  plan  up¬ 
dated,  complete,  and  coordinated  with  all  con¬ 
cerned  management  personnel  may  have  a 
negative  impact  on  the  program. 

3.21  MANAGING  PROBLEMS  IN  THE 
DEPLOYMENT  PROCESS 

Unreported  and  uncorrected  deploy¬ 
ment  problems  can  seriously  disrupt  the  proc¬ 
ess. 

3.22  DELAYED  POST  PRODUCTION 
SUPPORT  (PPS)  PLANNING 

Continued  support  of  the  material  sys¬ 
tem  by  the  industrial  base  existing  in  the  post- 


A—  1 1 


production  time  frame  may  not  be  economi¬ 
cally  feasible. 


3.23  ACCELERATED  ACQUISITIONS 

Lead  times  for  delivery  of  non-develop- 
mental  items  can  be  extremely  short,  particu¬ 
larly  for  in-stock  commercial  items.  This  poses 
a  substantial  risk  of  deployment  with  incom¬ 
plete  or  inadequate  logistic  support  and  atten¬ 
dant  degraded  readiness. 


3.24  CONFIGURATION  CONTROL  OF 

COMMERCIAL  ITEMS 

The  Government  does  not  control  the 
configuration  of  items  procured  from  the  com¬ 
mercial  marketplace.  This  presents  two  poten¬ 
tial  risks: 

•  Subsequent  competitive 
procurement  of  the  end 
item  may  lead  to  a  total  dif¬ 
ferent  internal  configura¬ 
tion  with  different  support 
requirements. 

•  There  is  no  automatic  guar¬ 
antee  that  original  com¬ 
mercial  suppliers  will  con¬ 
tinue  to  manufacture  spares 
and  repair  parts  to  fit  the 
Government’s  configura¬ 
tion. 


3.25  INADEQUATE  COORDINATION 

The  Government  does  not  control  the 
configuration  of  items  procured  from  the  com¬ 


mercial  marketplace.  This  presents  two  poten¬ 
tial  risks: 

•  Incomplete  or  inadequate 
logistic  support  at  the  time 
of  initial  deployment 

•  A  decision  by  one  or  more 
Services  to  go  it  alone  with 
ILS  planning  and  develop¬ 
ment  of  Service-unique  lo¬ 
gistics  support 

•  Loss  of  the  economies  of 
scale  that  can  be  gained  by 
joint  ILS  performance. 

Ref.  1  DSMC  Integrated  Logistics  Sup¬ 
port  Guide  Extracts. 
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1.  GENERAL 

Two  important  drivers  of  acquisition 
policy  regulations  are  DoD  Directive  5000.1 
and  DOD  Instruction  5000.2.  Both  were  re¬ 
cently  updated  and  released  to  the  Services 
(September  1,  1987),  and  there  has  not  been 
sufficient  time  for  the  Services  to  align  their  ac¬ 
quisition  policy  directives. 

2.  POLICY  DIRECTIVES 

This  section  briefly  summarizes  the 
listed  directives  and  regulations  of  OMB, 
DOD,  and  the  Services  pertaining  to  risk  man¬ 
agement.  Figure  C-l  shows  a  hierarchy  of  ac¬ 
quisition  risk  policy. 

a.  Office  of  Management  and  Budget 
OMB  Circular  A-109.  Major  System  Acquisi¬ 
tion  (5  April  1976) 

Para  7.  “Each  agency  acquiring  major 
systems  should ...  tailor  an  acquisition  strategy 
for  each  program...  The  strategy  should  typi¬ 
cally  include  ...  methods  for  analyzing  and 
evaluating  contractor  and  Government  risks.” 

b.  Department  of  Defense 

DOD  Directive  (DODD)  5000.1.  Major  and 
Non-Major  Defense  Acquisition  Programs  (Sep¬ 
tember  1,  1987). 

Para  C.6.a.  “A  DOD  acquisition  pro¬ 
gram  may  be  designated  as  a  major  defense  ac¬ 
quisition  program  because  of  (among  other 
factors)  development  risk.” 


Para  D.3.  “Acquisition  phases  are  to  be 
tailored  to  minimize  acquisition  time  and 
life-cycle  costs,  consistent  with  the  urgency  of 
need  and  degree  of  technical  risk  involved.  ” 

Para  D.7.  “Affordability,  which  is  a 
function  of  cost,  priority,  and  availability  of  fis¬ 
cal  and  manpower  resources,  shall  be  consid¬ 
ered  at  every  decision  milestone  and  during  the 
Planning,  Programming,  and  Budgeting  Sys¬ 
tem  (PPBS)  proc- 

Para  D.8.b.  “  DOD  Components  will 
seek  a  balance  between  development  and  pro¬ 
duction  risk  and  the  risk  associated  with  not 
countering  the  threat.” 

Para  D.8.C.  “DOD  Components  will  es¬ 
timate  program,  budget,  and  fund  acquisition 
programs  realistically.” 

Para  D.9.a.  “During  early  stages  of  de¬ 
velopment,  studies  shall  be  conducted  to  iden¬ 
tify  trade-offs  between  cost  and  Hierarchy  of 
Acquisition  Policy  performance  requirements 
and  to  assess  technological  risk  and  identify 
cost  drivers  and  producibility  factors  associ¬ 
ated  with  new  technology.” 

Para  D.9.b.  “Commensurate  with  risk, 
developing  separate  alternatives  in  high-risk 
areas  (among  other  strategies)  shall  be  consid¬ 
ered.” 

Para  D.9.e.  “Logistics  supportability  re¬ 
quirements  shall  receive  emphasis  comparable 
to  cost,  schedule,  and  performance  require¬ 
ments.” 
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Figure  C-l  Hierarchy  of  Acquisition  Risk  Policy 


Para  D.9.f.  “Contract  type  shall  be  con¬ 
sistent  with  all  program  characteristics  includ¬ 
ing  risk.  For  research  and  development  phases, 
a  cost-reimbursable  contract  sensibly  allocates 
program  risk  between  the  contracting  parties. 

DOD  Directive  5000.3.  Test  and  Evaluation 
(March  12, 1986) 

Para  C.l.c.  “Test  and  evaluation  will 
begin  as  soon  as  possible  in  the  system  acquisi¬ 
tion  process  to  reduce  acquisition  risks...” 

Para  7c.  “General  provisions  of  soft¬ 
ware  testing  include  ...  testing  of  software  to 
achieve  a  balanced  risk  with  the  hardware.” 

Para  C.4.d.  “Before  Milestone  II,  T&E 
shall  identify  the  preferred  technical  ap¬ 
proach,  the  technical  risks,  and  the  feasible  so¬ 
lutions.  ” 

DOD  5000.3-M-l  Test  and  Evaluation  Master 
Plan  (TEMP)  Guidelines  Appendix  A  Defini - 
lions  "Acquisition  Risk.  The  uncertainty  that 
some  element  of  an  acquisition  program  will 
produce  an  unintended  result  with  adverse  im¬ 
pact  on  system  effectiveness,  suitability,  cost, 
program  schedule,  or  availability  for  deploy¬ 
ment.” 

DOD  Directive  5000.4.  OSD  Cost  Analysis  Im¬ 
provement  Group  (October  30,  1980) 

Para  2.e.  “The  OSD  CAIG  shall  develop 
methods  of  formulating  cost  uncertainty  and 
cost  risk  information  into  the  acquisition  proc¬ 
ess.” 


Para  C.5.  “Quantifying  uncertainty  by 
using  frequency  distributions  of  cost  ranges  is 
desired.  Probability  distributions  and  assump¬ 
tions  should  be  forwarded  with  range  esti¬ 
mates.” 

Para  C.7.  “Sensitivity  analysis  of  cost 
should  include  technical  risks.” 

DOD  Directive  5000.29.  Management  of  Com¬ 
puter  Resources  in  Major  Defense  Systems  (April 
26, 1976) 

“B.  Requirements  Validation  and  Risk 
Analysis 

3.  Risk  analysis,  preliminary  design, 
hardware/software  integration  methodology, 
external  interface  control,  security  features 
(DoD  Directive  5200.28,  reference  (f)).  and 
life  cycle  system  planning  shall  be  included  in 
the  review  (DSARC  II).” 

DODI  5000.2.  Defense  Acquisition  Program 
Procedures  (September  1,  1987) 

Para  3.b.  “Defense  Acquisition  Board 
deliberations  are  (among  other  considera¬ 
tions)  program  risk  versus  benefit  of  added 
military  capability  and  procurement  strategy 
appropriate  to  program  cost  and  risk  assess¬ 
ments.” 

Enclosure  (3)  Mission-Needed  Statement 
(MNS)  Format 

5.  Technology  Involved.  “For  known  alterna¬ 
tives,  discuss  maturity  of  the  technology 
planned  for  the  selected  acquisition  design  and 
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manufacturing  processes,  with  particular  em¬ 
phasis  ri  remaining  areas  of  risk.” 

Enclosure  (4)  System  Concept  Paper  (SCP)  and 
D  cision  Coordinating  Paper  (DCP)  Formats 

9.  Technology  Risks  of  Selected  Alternatives. 
“For  Milestone  II,  discuss  test  and  evaluation 
results  that  show  all  significant  risk  areas  have 
been  resolved.” 

D0D1  5000.38.  Production  Readiness  Reviews 
(January  24,  1979) 

Para  A.2.  “The  objective  of  a  PRR 
(Program  Readiness  Review)  is  to  verity  that 
the  ptoduction,  design,  planning,  and  associ¬ 
ated  preparations  for  a  system  have  progressed 
to  the  point  where  a  production  commitment 
can  be  made  without  incurring  unacceptable 
risks  or  breaching  thresholds  of  schedule,  per¬ 
formance,  cost,  or  other  established  criteria.” 

Para  E.4.  “The  DPESO  (DOD  Product 
Engineering  Services  Office)  independent  pro¬ 
duction  readiness  assessment  will  consist  of 
objective  conclusions  based  on  the  findings  of 
the  PRR  and  other  investigations.  This  assess¬ 
ment  will  identify  potential  problem  area, 
which  constitute  production,  cost,  or  schedule 
risks.  Each  risk  will  be  expressed  in  terms  of  its 
relative  magnitude,  and  potential  consequences." 
[emphasis  supplied] 

DODl  704 1 .3.  Economic  Analyses  and  Program 
Evaluation  for  Resource  Management  (October 
19,  1972) 


Enclosure  (2) 

Para  B.7.  "Risk! Uncertainty  Analysis. 
Risk  assessments  will  be  made  to  determine  the 
expectation  or  probability  that  program/project 
objectives  will  be  realized  by  following  a  specific 
course  of  action  with  constraints  of  lime,  cost, 
and  technical  performance,  [emphasis  sup¬ 
plied]  Actual  costs  and  outputs  of  many  DOD 
projects  differ  from  those  expected  at  the  time 
of  decision.  For  those  cases,  and  in  particular 
for  major  weapon  systems  covered  by  a  Se- 
.ected  Acquisition  Review  Report  or  subject  to 
review  by  the  Defense  System  Acquisition  Re¬ 
view  Council  (DSARC).  the  impact  which 
could  result  from  this  variability  should  be 
evaluated.” 

Para  B.7.a.  “Independent  parametric 
cost  estimates  can  provide  an  early  test  of  the 
reasonableness  of  cost  estimates.  Independent 
parametric  cost  estimates  will  be  made  at  key 
decision  points  for  major  weapon  systems,  e.g.. 
during  concept  formulation  and  prior  to  mak¬ 
ing  major  commitments  of  funds  for  develop¬ 
ment  and  production.  These  estimates 
generally  consider  cost  at  high  levels  of  aggre¬ 
gation  and  are  predicated  on  actual  historical 
costs  encountered  in  like  or  similar  programs. 
As  such,  they  incorporate  costs  for  expected 
uncertainties  on  the  average.  (1)  Costs  should 
be  derived  by  parametric  techniques  and  ex¬ 
pressed  as  feasible  rmgesin  terms  of  the  pa¬ 
rameters  which  drive  them.  It  is  most  important 
that  estimates  be  presented  as  cost  ranges  re¬ 
lated  to  the  probable  values  of  system  parameters, 
characteristics,  or  attributes  which  are  deter¬ 
mined  by  costs,  [emphasis  supplied]  (2)  These 


estimates  will  be  available  for  each  DSARC 
review.  Parametric  estimates  will  be  derived  in¬ 
dependent  of  functional,  program  manager,  or 
contractor  influence.  (3)  When  the  independ¬ 
ent  parametric  cost  estimate  differs  from  the 
program  manager’s  current  estimate,  the  latter 
estimate  will  be  used  for  economic  analysis/ 
program  evaluations.  Once  a  program  estimate 
is  established  as  a  baseline,  a  program/project 
manager  will  manage  his  program  within  that 
limitation.  (4)  The  program  manager’s  current 
estimate  will  be  an  assessment  of  the  ultimate 
cost  expected  for  a  program/project  including 
undefinitized  contingencies,  [emphasis  sup¬ 
plied]  As  such,  the  program  manager’s  current 
estimate  should  be  relatively  stable  over  long 
periods  of  time  and  not  change  with  small  in¬ 
cremental  changes  to  the  approved  program, 
funding  changes,  or  financial  fluctuations.  To 
the  extent  possible ,  schedules  and  funding  should 
be  structured  to  accommodate  program  uncer¬ 
tainties  and  unforeseen  problems."  [emphasis 
supplied] 

Para  B.7.b.  “Special  degrees  of  risk/un¬ 
certainty  associated  with  a  particular  program/ 
project,  may  be  pointed  out  quantitatively  in  an 
analysis  and  used  for  program  review  purposes. 
Probability  estimates  can  be  developed  by  test¬ 
ing  the  sensitivity  of  key  variables  on  esti¬ 
mated  costs  and  performance.  The  probability 
that  each  of  the  possible  cost  or  output  estimates 
may  be  realized  should  be  discussed  narratively 
when  there  is  no  basis  for  a  quantitative  esti¬ 
mate."  [emphasis  supplied] 

Para  b.7.c.  “Estimates  will  be  expressed 
in  terms  o*’  performance  thresholds,  goals,  or 


ranges.  Program/project  estimates  will  include 
the  limits  within  which  ultimate  program  cost 
and  technical  performance  is  expected  to  fall.” 

3.  SERVICE  REQUIREMENTS 
a.  U.S.  Army 

The  following  Army  directives  require  or  im¬ 
ply  a  need  for  risk  assessment  as  shown  by 
excerpts  and  editorial  condensations. 

Department  of  the  Army  Pamphlet  (DA- 
Pam)  11-2.  Research  and  Development  Cost 
Guide  for  Army  Materiel  Systems  (May  1976). 

Para  3.5.  Range  Versus  Point  Estimates. 

a.  ‘The  use  of  a  point  estimate  does  not  retted 
the  uncertainty  associated  with  the  estimate.  It 
aiso  implies  that  it  is  a  precise  cost.  For  these 
reasons,  a  range  of  costs  should  be  provided 
based  on  the  inherent  cost  estimating  uncer¬ 
tainty.  The  level  at  which  the  ranges  can  be  pro¬ 
vided  is  dependent  upon  the  level  at  which  the 
costs  are  estimated.  Within  the  limitations  im¬ 
posed  by  the  database  and  cost  estimating  ap¬ 
proach  employed,  ranges  should  be  presented 
at  the  highest  aggregate  level,” 

b.  "In  addition,  an  analysis  should  be  made  of 
the  sensitivity  of  projected  costs  to  all  critical 
assumptions.  This  should  include  factors  such 
as  the  impact  of  changes  in  performance  char¬ 
acteristics,  changes  in  configuration  to  meet 
performance  requirements,  schedule  alterna¬ 
tives,  and  alternative  production  processes.” 


DA-Pam  11-3.  Investment  Cost  Guide  for 
Army  Materiel  Systems  (April  1976)  Exactly 
the  same  words  as  above. 

DA-Pam  11-4.  Operating  and  Support  Cost 
Guide  for  Army  Materiel  Systems  (April  1976). 
Exactly  the  same  words  as  above. 

Development  Acquisition  and  Readiness 
Command  Regulation  (DARCOM-R)  11-1. 
Systems  Analysis 

Para  4.d.  “...RA  and  DR  A  are  applied  to 
alternative  courses  of  action  and  permit  struc¬ 
turing  models  that  address  the  uncertainty  of 
cost,  schedule,  and  performance  of  systems.” 
[RA  and  DRAmean  risk  analysis  and  decision 
risk  analysis,  respectively.] 

Para  5.c.  “An  IE  and  DR  A  will  be  com¬ 
pleted  prior  to  each  decision  milestone  in  ma¬ 
jor  programs  which  will  involve  ....  (ASARC) 
or ....  (DSARC)  proceedings,  or  in  non-major 
programs  for  which  DA  has  retained  in-proc¬ 
ess  review  (IPR)  approval  authority.  For  non¬ 
major  systems,  an  ASARC  will  be  completed 
prior  to  each  IPR  “unless  it  is  clear  that  no  ap¬ 
preciable  time,  cost,  or  performance  risk  is  as¬ 
sociated  with  the  decision.”  [IE  means 
Independent  Estimate;  ASARC,  Army  Sys¬ 
tems  Acquisition  Review  Council;  DSARC, 
Defense  Systems  Acquisition  Review  Council; 
DA,  Department  of  the  Army] 

Para  6.d.  “Each  commander  of  an  R&D 
or  MR  Command  will:  ...3)  ensure  that  IE’s  and 
DRA’s  are  initiated  for  the  decision  points  de¬ 


scribed  in  paragraph  5c.”  [MR  means  Materiel 
Readiness.] 

Appendix  D  -  Decision  Risk  Analysis 
Guidelines 

“1.  a.  Define  the  problem 

b.  Establish  the  decision¬ 
maker’s  preferences  for 
trade-offs  between  cost, 
schedule,  and/or  perform¬ 
ance 

2.  Establish  the  alternatives 

3.  Define  the  events 

4.  Collect  the  data 

5.  Determine  the  program 
risks 

6.  Select  the  best  alternatives 

7.  Perform  sensitivity  analysis 

8.  Communicate  the  results.” 

DARCOM-R  11-27.  Life  Cycle  Management 
of  DARCOM  Materiel  Chapter  3.  Section  11  - 
Procedures 

Para  3~8.b.  “The  justifications  for  by¬ 
passing  activities  and  events  are ...  (2)  That  the 
risk  for  omitting  the  actions  is  reasonable  when 
considering  the  savings  of  time  and  resources.” 

Para  3-8.c.  “When  requesting  the 
shortening  of  schedules,  the  PM  will  request 
DARCOM  approval  and  submit  a  statement 
including  the  assessment  of  risks  incurred  in 
shortened  plan  compared  to  base  plan.” 

AR-11-28.  Economic  Analysis  and  Program 
Evaluation  for  Resource  Management 
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Chapter  1 

Para  l-6.p.  “Where  costs  for  research 
and  development  represent  a  significant  por¬ 
tion  of  total  program  cost,  the  decision  to  con¬ 
duct  research  will  be  supported  by  an  economic 
analysis  which  identifies  potential  follow-on 
cost  savings  resulting  from  the  research  and 
development,  degree  of  risk  or  uncertainty  in 
achieving  results,  availability  of  resources,  as¬ 
sessment  of  current  technology,  and  identifica¬ 
tion  of  constraints.” 

Chapter  2 

Para  2-2. b.  “The  structure  of  analysis 
will  also  contain,  when  appropriate,  an  assess¬ 
ment  of  the  relative  risk  or  uncertainty  of  suc¬ 
cess  associated  with  each  of  the  alternatives 
considered,  including  the  status  quo  when  ap¬ 
plicable.” 

AR-15-14.  Boards,  Commissions,  and  Com¬ 
mittees  -  Systems  Acquisition  Review  Council 
Procedures 

Page  4-3.  This  paragraph  states  that  risk 
analysis  will  be  presented  to  HQDA  two 
months  before  ASARC. 

AR-70-1.  Systems  Acquisition  Policy  and  Pro¬ 
cedures  (12  Nov  1986) 

Para  l-5.c  Objectives  of  Army  research, 
development  and  acquisition  are  ...  “Achieve 
appropriate  balance  between  need  for  low  risk 
evolutionary  development  and  more  vision¬ 
ary,  leap-ahead  effort  required  to  maintain 
technological  superiority.” 


Para  4.5-z  Criteria  for  entering  the 
Dem/Val  phase  are  ...  “Production  feasibility 
has  been  addressed  and  areas  of  production 
risk  evaluated  including  the  availability  and 
completeness  of  TDP.  Manufacturing  technol¬ 
ogy  needed  to  reduce  production  risks  to  ac¬ 
ceptable  levels  have  been  identified.  Costs 
and  technical  plans  for  meeting  alternative 
surge  capacities  have  been  prepared  and  con¬ 
straints  to  attaining  expanded  production  are 
identified.” 

Para4~6.q.(3)  Criteria  for  entering  the 
FSD  phase  are  ...  “Production  risk  has  been 
determined  acceptable.  Requirements  for  long 
lead-time  procurements,  initial  production  fa¬ 
cilities  (IPF),  and  limited  production  have  been 
identified  and  evaluated  considering  planned 
production  and  expanded  production  require¬ 
ments  for  surge  and  mobilization.  The  FSD 
Phase  includes  PEP  provisions  to  attain 
producibility,  using  cost  effective  manufactur¬ 
ing  methods  and  processes.  Manufacturing 
methods  and  technology  (MMT)  deficiencies 
have  been  addressed,  and  included  in  the  PEP 
program  summary  requirements.” 

Para  4-7.i.  Ciiteria  for  entering  the  pro¬ 
duction  and  development  phase  are  ...  “PEP 
has  been  conducted,  production  proveout 
(product,  process,  and  facilities)  has  been  suc¬ 
cessful  and  the  chosen  surge  capacity  has  been 
measured  by  extrapolation  from  actual  produc¬ 
tion;  and  economic  timely  producibility  has 
been  determined.  Production  readiness  review 
and  assessment  has  been  completed;  produc¬ 
tion  risks  have  been  reduced  to  acceptable  lev¬ 
els;  and  constraints  and  remedies  to  increased 
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production  beyond  planned  surge  level  are 
identified.” 

Para  4-12.  c.  (4).  “An  assessment  of 
RAM,  RAM-driven  O&S  costs  and  product  as¬ 
surance  issues  will  be  provided.  For  each  unre¬ 
solved  RAM  or  quality  assurance  issue,  a 
proposed  resolution  shall  be  provided.  This 
resolution  will  include  assessment  of  risk, 
RAM-driven  O&S  impact  on  quantitative 
RAM  parameters.” 

Para  5-l.c.(2).  The  Acquisition  Strat¬ 
egy  does  the  following  ...  “Identifies  potential 
risks  and  plans  to  reduce  or  eliminate  risks.” 

Para  6-2.  “Under  AR  71-9,  all  require¬ 
ments  documents  will  include  provisions  for 
P3I.  The  drivers  for  these  provisions  are  techni¬ 
cal  risk  and  threat,  and  O&S  cost. 

(1)  Where  an  early  deployment  capabil¬ 
ity  is  required,  but  one  or  more  key  component 
subsystems  are  judged  to  be  technically  risky 
or  the  technology  is  judged  to  require  consid¬ 
erably  more  maturation  than  the  rest  of  system, 
then  the  requirement  documentation  should 
structure  acceptable  performance  criteria  or  a 
phased  deployment  capability  (e.g.,  initial,  in¬ 
terim,  and  objective).” 

Para  7-2.a.  “The  Army  Streamlined  Ac¬ 
quisition  Process  (ASAP)  is  essentially  a  syner¬ 
gistic  combination  of  common  sense  measures, 
derived  from  lessons  learned  in  a  variety  of  ac¬ 
quisition  programs,  to  achieve  the  ‘surest  and 
shortest'  path  for  low  risk  developments  while 
eliminating  the  need  for  case-by-case  excep¬ 
tions  to  the  traditional  acquisition  process..." 


Para  7-2.  F.  (9).  “Minimum  essential  test 
and  evaluation  necessaty  to  identify  the  best 
technical  approach  to  include  identification  of 
technical  risk  and  feasible  solutions.  Test  data 
shall  support  projection  of  realistic  program 
performance  and  suitability  thresholds.  Mod¬ 
eling  and  simulation  is  encouraged  to  ensure 
availability  of  operational  effectiveness  and 
suitability  projections.  The  MS  II  decision  will 
be  preceded  by  sufficient  T&E  to  reduce  risk 
before  resources  are  committed  to  FSD.” 

Para  l-7.o.  “Technical  uncertainty  will 
be  continuously  assessed.  Progressive  commit¬ 
ments  of  resources  will  be  made  only  when 
confidence  in  program  outcome  is  sufficiently 
high  to  warrant  going  ahead.” 

Para  2-2.a.  In  conceptual  phase,  “criti¬ 
cal  technical  issues,  operational  issues,  and  lo¬ 
gistical  support  problems  are  identified  for 
resolution  in  subsequent  phases  in  order  to 
minimize  future  development  risks.” 

Para  2-15.a.  ‘Test  and  evaluation  will 
be  conducted  as  early  as  possible  and 
throughout  the  material  acquisition  process  to 
reduce  acquisition  risks....” 

Para  2-15.c.(4).  DT  will  be  used  to 
“demonstrate,  during  the  Full  Scale  Develop¬ 
ment  Phase  and  prior  to  the  first  major  produc¬ 
tion  decision,  that  the  DT  accomplished  is 
adequate  to  insure  that  engineering  is  reason¬ 
ably  complete;  that  all  significant  design  prob¬ 
lems  ...  have  been  identified;  and  that  solutions 
to  the  above  problems  are  in  hand.” 

Para  4-la.  “Department  of  the  Army 
policy  for  advanced,  engineering,  and  opera- 
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tional  systems  development  is  to  —  (1)  Con¬ 
duct  system  advanced  development  in 
promising  areas  using  either  single  or  competi¬ 
tive  approaches  in  order  to  resolve  key  techni¬ 
cal,  cost  and/or  schedule  uncertainties  before 
entering  Full-Scale  Development  Phase.  Such 
efforts  should  be  accomplished  with  low-level 
program  and  full  realization  of  technical  risks.” 

Para  4~lm.  “Program  sufficient  funds 
to  provide  for  the  technical  uncertainty  inher¬ 
ent  in  the  development  effort.” 

Para  4-laf.  “Give  consideration  to  re¬ 
quiring  development  contractors  to  provide 
second  sources  for  high  technical  risk  subsys¬ 
tems/components,  whether  or  not  the  devel¬ 
opment  contract  is  sole  source  or  competitive.” 

AR  70-10.  Research  and  Development,  Test  and 
Evaluation  During  Development  and  Acquisi¬ 
tion  of  Materiel  (29  August  1975). 

Para  1-4. (3).  “During  the  full-scale  de¬ 
velopment  phase  and  prior  to  the  first  major 
production  decision,  the  DT  (Development 
Test)  accomplished  will  be  sufficiently  ade¬ 
quate  to  insure  ...  that  all  significant  design 
problems  (including  compatibility,  inter¬ 
operability,  safety,  Reliability,  Availability, 
and  Maintainability  (RAM),  and  supportability 
considerations)  have  been  identified,  and  that 
solutions  to  the  above  problems  are  in  hand.” 

Para  2-4. c.  (5).  “As  the  development  cy¬ 
cle  continues  into  DT/OT  [Development  Test/ 
Operational  Test]  II,  the  uncertainty  in  these 
estimates  should  be  reduced,  and,  by  the  com¬ 


pletion  of  DT/OT  III,  sufficient  testing  should 
have  been  accomplished  so  that  the  uncer¬ 
tainty  in  estimating  the  final  system  perform¬ 
ance  will  be  relatively  small, ....” 

Para  2-5.e.  “EDT  (Engineering  Devel¬ 
opment  Test)  is  conducted  by  the  contractor 
and/or  the  material  developer  with  the  primary 
objective  of  influencing  material  design.”  ... 
“The  purposes  of  EDT  are  to  ...(3)  Eliminate 
as  many  technical  and  design  risks  as  possible 
or  to  determine  the  extent  to  which  they  are 
manageable.” 

AR  71-9.  Materiel  Objectives  and  Require¬ 
ments,  Para  4-18  “The  provisions  of  P3I  will  be 
considered  in  all  developmental  material  pro¬ 
grams  and  documented  in  requirements  docu¬ 
ments  as  appropriate.  The  P3I  is  a  strategy  that 
offers  an  alternative  that  minimizes  techno¬ 
logical  risk  and  consciously  insures  advanced 
technology  through  planned  upgrades  of  those 
deployed  systems  or  subsystems  that  offer  the 
greatest  benefits.  In  this  manner,  the  lead- 
time  to  field  technological  advances  can  be 
shortened  while  an  aggressive  scheduling  of 
fielded  performance  improvements  can  be  ex¬ 
pected  during  the  service  life  of  the  system." 

Para  1-5.  “Test  and  Evaluation  will  be¬ 
gin  as  early  as  possible  in  the  acquisition  cycle 
and  will  be  conducted  throughout  the  system 
acquisition  process  as  necessary  to  assess  ac¬ 
quisition  risks  ...” 

Para  3-4.k.  “Army  R&D  organizations 
are  to  take  the  following  specific  actions  with 
respect  to  the  STOG  (Science  and  Technology 
Objectives  Guide):  (1)  Perform  assessment 
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components,  or  system  types,  depending  upon 
the  detail  available.  In  such  cases,  risk  factors 
will  be  constructed  judgmentally  in  full  consid¬ 
eration  of  the  engineering,  producibility,  and 
budgetary  aspects  of  the  program.  Specific  con¬ 
siderations  to  be  included  in  this  judgment 
are: 

(a)  Whether  the  program  requires  the  devel¬ 
opment  of  an  item  not  directly  supported  as 
feasible  by  existing  technology. 

(b)  Whether  the  program  requires  the  devel¬ 
opment  of  an  item  substantially  different  from 
those  previously  developed. 

(c)  Whether  major  integration  effort  will  be 
necessary  even  though  individual  components 
may  in  themselves  be  considered  to  involve  low 
risk.” 

Para  5.b.  “TRACE  computation.  The 
risk  factors  will  be  multiplied  by  the  engineer¬ 
ing  cost  estimate  at  the  appropriate  level  of  the 
WBS.  The  appropriate  level  will  depend  not 
only  on  the  level  of  design  detail  available, 
but  also  on  the  degree  of  component  and  sub¬ 
system  interaction.  In  those  circumstances 
where  a  design  change  of  a  given  component  or 
subsystem  appears  likely  to  propagate  and 
cause  a  design  change  of  a  related  component 
or  subsystem,  a  higher  level  of  aggregation  will 
also  be  required  to  maintain  statistical  validity 
of  the  overall  estimate  by  including  these  inter¬ 
dependent  effects.  The  risk  factors,  when  ap¬ 
plied  at  the  appropriate  level  of  the  WBS  as 


explained  above,  can  be  statistically  combined 
to  produce  the  TRACE.” 

Para  6. a.  “The  costs  of  specific  program 
work  scheduled  for  accomplishment  during  a 
particular  year  will  be  estimated  using  the 
TRACE  methodology.  The  TRACE  thus  com¬ 
piled  for  the  program  ‘work  year’  will  be  the 
amount  submitted  to  OSD  and  the  Congress  as 
required  for  the  program  for  that  year.  The 
funds  representing  the  difference  between  the 
TRACE  and  the  engineering  cost  estimate  will 
not  be  carried  in  a  separate  category,  but 
rather  will  be  allocated  to  the  various  tasks  to 
which  the  funds  will  most  likely  be  applied." 

Para  6.b.  “To  allow  for  the  possibility  of 
cost  savings  to  allow  more  precise  managerial 
control  of  funds  appropriated  for  program  exe¬ 
cution  during  a  budget  year,  only  that  amount 
reflecting  the  basic  (engineering)  cost  esti¬ 
mated  for  that  year  (i.e.,  the  engineering  cost 
estimate  consisting  of  the  work  costs  prior  to 
multiplication  by  the  respective  risk  factors) 
will  be  released  initially  to  the  manager  for  pro¬ 
gram  execution.  The  remainder  of  the  appro¬ 
priated  program  funds  will  be  held  in  deferral 
by  the  DCSRDA  and  released  to  the  manager 
only  upon  request  and  approval  of  a  justified 
need.  Program  funds  (obligated  authority)  not 
required  in  the  cut  rent  year  program  will  be 
considered  for  designation  to  the  Congress  as  a 
means  to  reduce  the  requirement  for  new 
obligational  authority  is  the  subsequent  year’s 
budget.  Other  use  of  such  unneeded  funds  may 
be  authorized  by  the  DCSRDA,  as  appropri¬ 
ate.” 
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Letter  of  Instruction  (LOI)  for  Implementation 
of  the  Total  Risk  Assessing  Cost  Estimate  for 
Production  (TRACE-P)  (6  October  1982). 

Para  4.b.  “The  TRACE-P  estimate  will 
include  consideration  of  the  risks  in  the  follow¬ 
ing  categories;  these  are  explained  at  Enclo¬ 
sure  1. 

(1)  Threat  Uncertainty 

(2)  Management 

(3)  Materials/Purchased  Parts 

(4)  Facilities/Equipment 

(5)  Labor 

(6)  Design  Changes 

(7)  Producibility 

(8)  Performance.” 

Para  4.c.  “Specifically  excluded  from 
the  estimating  of  TRACE-P  expected  risk 
costs  are  possible  increases  that  may  result 
from  one  or  more  of  the  following  causes: 

(1)  Quantity  changes 

(2)  Performance  improvement 
to  meet  an  increase  threat 

(3)  Poor  management 

(4)  Inadequate  funding  in  the 
early  years 

(5)  Unknown  unknowns.” 

Para  4.g.  “TRACE-P  funds  will  be  held 
in  deferral  by  the  DCSRD  A  and  released  to  the 
program/project/product  manager  only  upon 
request  and  approval  of  a  justified  need.” 

RDA  Cost  Realism  -  Future  Development  Pro¬ 
grams  (12  July  1974)  DASA  Letter 


“Our  estimate  should  be  unbiased  so  that  we 
have  about  an  even  chance  of  either  going 
over  or  under  it.” 

“It  is  submitted  that  cost  overruns  will  continue  to 
be  a  way  of  life  until  adequate  recognition  is  given 
to  the  impact  of  program  uncertainty  in  estimat¬ 
ing  costs ”  [emphasis  in  original]. 

“...  it  remains  the  fundamental  nature  of 
RDT&E  ...  to  involve  the  unknown.  These  un¬ 
knowns  invariably  lead  to  cost  requirements 
which  cannot  be  individually  foreseen  at  the 
outset  of  a  development  —  yet  their  cumula¬ 
tive  impact  can  be  seen  in  retrospect  with  all 
the  assuredness  of  the  laws  of  probability" 
[emphasis  in  original]. 

“The  provision  of  flexibility  in  the  funding  plan 
baseline-cost  estimates  should  reflect  these  prob¬ 
able  additional  costs." 

Total  Risk  Assessing  Cost  Estimate  (TRACE) 
Deferrals  DRCEPC  letter  (17  April  1978). 

Para  2.  “A  program  TRACE  refers  to  its 
total  expected  RDT&E  costs  as  agreed  to  by 
the  ASARC/DSARC.  The  definition  applies 
both  to  annual  costs  and  development  costs. 
Funds  in  excess  of  the  baseline  or  engineering 
costs  for  a  particular  fiscal  year  are  deferred  at 
the  beginning  of  that  year  by  HQDA 
(DCSRDA)  pending  the  occurrence  of  pre¬ 
dicted  (but  unprogrammed)  events  upon  which 
the  funds  were  based.  These  funds  will  be  re¬ 
leased  to  the  Project  Manager  (PM)  upon; 
demonstration  that  they  are  necessary  to  ou 
set  the  cost  of  such  events.  Ifa  program  adjust- 
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ment  is  made  during  the  programming  and 
budgeting  cycle,  the  entire  scope  of  the  pro¬ 
gram  should  be  revaluated  and  the  risk  factor 
recomputed.  TRACE  funds  identified  for  de¬ 
ferral  in  the  outyear  should  not  be  reduced  in 
order  to  increase  the  baseline  portion  if  that 
adjustment  is  made  merely  to  offset  a  decre¬ 
ment  to  the  program  or  to  increase  its  scope.” 

Para  3.  “The  funds  released  i'rom 
HQDAare  expected  to  be  adequate  for  execu¬ 
tion  of  the  known  estimated  engineering  costs. 
It  is  DARCOM  policy  to  expect  that  the  PM 
manage  his  total  program  with  the  funds 
authorized.  Risk  contingency  (TRACE  defer¬ 
ral)  funds  will  be  released  only  if  technical/  de¬ 
sign  problems  and/or  unexpected  delays 
materialize,  and  this  fact  is  completely  dem¬ 
onstrated  in  the  release  request.  Release  re¬ 
quires  both  DCSRDA  and  ASA  (R&D) 
approval.” 

Para  4.  “The  PM  can  indicate  at  any 
time  that  TRACE  deferral  funds  will  not  be 
needed.  If  deferral  has  not  been  released  by  the 
fifth  quarter  of  availability,  the  PM  will  be 
given  the  opportunity  to  justify  retention  of 
such  deferrals  considering  that  the  work  upon 
which  they  were  based  may  have  continued 
into  the  second  year.  Otherwise,  disposition  of 
the  funds  will  be  determined  by  the  DCSRDA 
in  coordination  with  the  ASA  (RL&LD)  and 
CDR  DARCOM.  The  PM  may  request  that 
the  unneeded  TRACE  deferral  funds  be  re¬ 
leased  for  expanded  scope  of  work  in  the  same 
weapons  systems;  however,  that  request  will 


be  evaluated  with  other  unfinanced  require¬ 
ments  in  other  programs.” 

6.  US  Air  Force 

The  following  Air  Force  Regulations 
pertaining  to  program  risk  are  briefly  summa¬ 
rized. 

Air  Force  Regulation  (AFR)  70-15.  Source  Se¬ 
lection  Policy  and  Procedures  (25  February 
1984) 

Para  3-4.  Assessment  of  Risk: 

a.  “Identification  and  assessment  of  the  risks 
associated  with  each  proposal  are  essential  The 
following  definitions  of  risk  should  be  used: 

(1)  HIGH  (H)  --  Likely  to  cause  significant 
serious  disruption  of  schedule,  increase  in  cost, 
or  degradation  of  performance  even  with  spe¬ 
cial  contractor  emphasis  and  close  government 
monitoring. 

(2)  MODERATE  (M)  —  Can  potentially  cause 
some  disruption  of  schedule,  increase  in  cost, 
or  degradation  of  performance.  However,  spe¬ 
cial  contractor  emphasis  and  close  government 
monitoring  will  probably  be  able  to  overcome 
difficulties. 

(3)  LOW  (L)  —  Has  little  potential  to  cause 
disruption  of  schedule,  increase  in  cost,  or 
degradation  of  performance.  Normal  contrac¬ 
tor  effort  and  normal  government  monitoring 
will  probably  be  able  to  overcome  difficulties. 

b.  The  acquisition  activity  or  program  office 
should  prepare  and  furnish  to  the  SSEB  an  in- 
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dependent  assessment  of  potential  risks  before 
receipt  of  proposals. 

c.  As  a  part  of  their  proposal,  offerors  are  re¬ 
quired  to  submit  a  risk  analysis  which  identifies 
risk  areas  and  the  recommended  approaches 
to  minimize  the  impact  of  those  risk  on  the 
overall  success  of  the  program. 

d.  The  risks  which  must  be  assessed  are  those 
associated  with  cost,  schedule,  and  perform¬ 
ance  or  technical  aspects  of  the  program.  Risks 
may  be  inherent  in  a  program  by  virtue  of  the 
program  objectives  relative  to  the  state  of  the 
art.  Risks  may  also  occur  as  a  result  of  a  par¬ 
ticular  technical  approach,  manufacturing 
plan,  the  selection  of  certain  materials,  proc¬ 
esses,  equipment,  etc.,  or  as  a  result  of  the  cost, 
schedule,  and  economic  impacts  associated 
with  these  approaches. 

e.  In  evaluating  risk,  the  evaluators  must  con¬ 
sider  the  program  office  assessment,  the  of¬ 
feror’s  assessment  and  make  an  independent 
judgment  of  the  probability  of  success,  the  im¬ 
pact  of  failure,  and  the  alternatives  available  to 
meet  the  requirements. 

f.  It  is  the  responsibility  of  the  technical  evalu¬ 
ation  teams  to  make  sure  that  the  cost  team  is 
informed  of  the  identified  risk  areas  and  the 
potential  for  cost  impact.” 

AFR  80-14.  Test  and  Evaluation  (3  November 
1986) 


Para  4.  “The  primary  purpose  of  Devel¬ 
opment  and  Operational  Test  and  Evaluations 
are  to  identify,  assess,  and  reduce  the  acquisi¬ 
tion  risk”  (among  other  purposes). 

Para  4c.  “After  Milestone  I,  Test  and 
Evaluation  is  conducted  to  identity  design 
risks”  (among  other  things). 

Para  13i.  “Major  objectives  of  Develop¬ 
ment  Test  and  Evaluation  are  to  assess  the 
technical  risk  and  evaluate  compliance  with 
the  specification.” 

Air  Force  Regulation  AFR  173-11.  In¬ 
dependent  Cost  Analysis  Program  (7  Oct  1986). 

Para  4.c(9).  Risk  and  Uncertainty 
Analysis.  An  explicit  assessment  of  program 
risk  is  included  as  part  of  each  ICA.  This  ap¬ 
plies  to  all  elements  of  cost,  including  O&S. 
When  possible,  this  risk  or  uncertainty  should 
be  quantified  in  dollars.  Risk  related  to 
parametric  and  critical  assumptions,  e.g.,  com¬ 
petition  savings,  technical  and  schedule  uncer¬ 
tainties,  improvement  curves,  factored  costs, 
and  cost- estimating  techniques,  etc.,  are  in¬ 
cluded. 

The  purpose  of  this  assessment  is  to 
identity  areas  of  cost  risk  and  place  the  ICA  in 
context  for  the  Air  Force  decision-maker.  The 
Air  Force  decision-maker  needs  to  know 
where  the  cost  sensitivities  are  in  a  program. 
As  a  minimum,  the  risk  or  uncertainty  analysis 
should  identify  the  high-risk  cost  elements  or 
cost-sensitive  assumptions,  e.g.,  weight,  com¬ 
petition  benefits,  cost-improvement  curve,  etc. 
The  analysis  should  also  show  the  probable 
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range  of  the  risk  parameters  and  the  associated 
costs  over  that  range. 

AFR  300-2.  Managing  the  USAF  Automated 
Data  Processing  Program  (24  April  1980). 

Para  3.1  (3).  “Management  will  evaluate 
known  risks.” 

AFR  800-3.  Engineering  for  Defense  Systems , 
(17  June  1977). 

Para  4.b.  [In  the  validation  phase]  “... 
certain  technical  aspects  may  need  to  be  inten¬ 
sified,  such  as  technical  and  cost  risk  reduction, 
obtaining  a  best  mix  of  technical  requirements, 
and  other  considerations  or  thresholds  as  may 
be  described  in  the  PMD.” 

Para  6.f.  The  AFSC  “programs  their  re¬ 
search  and  development  (R&D)  projects  to  de¬ 
velop  and  improve  systems  engineering 
methods  and  techniques  (system  cost  effective¬ 
ness,  risk  assessment,  technical  performance 
measurement,  etc.).” 

AFR  800-8.  Integrated  Logistics  Support  (1LS) 
Program  (25  June  1986) 

Para  6.b.(l).  “Ensures  acquisition  pro¬ 
gram  management  policy  and  guidance  em¬ 
phasizes  reliability,  maintainability,  and 
supportability  equal  to  cost,  schedule,  and  per¬ 
formance.” 

AFR  800-9.  Manufacturing  Management  Policy 
For  Air  Force  Contracts  (8  November  1983). 


Para  3.(10).a.  “Ensure  that  the  program 
manager  evaluates  the  relationship  between 
producibility,  manufacturing  risks,  productiv¬ 
ity,  ...and  probability  of  meeting  cost-related 
goals.” 

AFR  80G-14.  Life  Cycle  Management  of  Com¬ 
puter  Resources  in  Systems  (29  September 
1986). 

Para  3-12.  “The  program  manager,  with 
CRWG  assistance,  will  follow  the  principles  in 
attachment  5  to  identify,  assess,  and  control 
risk  associated  with  computer  resources." 
[CRWG  -  Computer  Resources  Working 
Group] 

Risk  Management ,  Attachment  5  (29 
September  1986). 

Para  A5-1.  “Risk  Management  for 
Computer  Resources.  In  most  development 
programs,  there  are  elements  that  pose  risks  to 
achievement  of  the  cost,  schedule,  support,  or 
performance  objectives  of  the  program.  His¬ 
torically,  development  of  computer  resources, 
especially  software,  has  been  one  of  the  high 
risk  elements.  Accordingly,  computer  resource 
development  efforts  need  to  apply  risk  man¬ 
agement  to:  identity  the  areas  that  introduce 
substantial  risk  to  program  objectives;  deter¬ 
mine  the  specific  causes  of  the  high  level  of 
risk;  eliminate  or  mitigate  the  causes  of  risk; 
establish  funding  reserves,  schedule  slack,  per¬ 
formance  margins,  and  contingency  plans  to  al¬ 
low  for  failure  of  original  plans;  and  monitor 
high  risk  areas  to  obtain  early  warning  of  fail¬ 
ures  and  allow  timely  activation  of  contingency 
plans.  Risk  management  efforts  must  be  de- 
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fined  early  in  the  program,  documented  in  a 
risk  management  plan,  and  adjusted  as  circum¬ 
stances  change.  Some  common  causes  of  high 
software  development  risk  and  possible  correc¬ 
tive  actions  are  listed  in  Table  A5-1.” 

c.  US  Navy 

The  following  Navy  directives  address  pro¬ 
gram  evaluation  including  some  mention  of 
risk  evaluation.  Excerpt  and  editorial  summa¬ 
ries  are  presented. 

Secretary  oi  the  Navy  Instruction  (SEC- 
NAVINST)  5000.  IB  System  Acquisition  (8 
April  1983). 

Para  5.g.  Management  Principles  and 
Objectives.  The  instruction  presents  as  a  man¬ 
agement  principle  “applying  established  or 
evolving  technology  having  a  high  probability 
of  success.  High  technical  risks  may  be  taken  if 
an  extraordinary  payoff  potential  can  be  dem¬ 
onstrated.” 

Para  6.  ‘ Acquisition  Categories.  A  pro¬ 
gram  is  a  candidate:  ACAT  IIS  designation  by 
SECN  AV, ...  if  it  is  a  special  SECN  AV  interest, 
...  because  of ...  a  history  of  technical,  cost,  and 
schedule  problems,”  or  “an  extraordinary  strat¬ 
egy  and/or  risks.” 

Para  8.  “ Decision  Milestones.  Mile¬ 
stones  and  phases  will  be  tailored  to  fit  each 
program  to  reduce  acquisition  time  and  cost, 
consistent  with  risk.” 

Para  8.b.  “Milestone  II.  It  should  be 
demonstrated  to  the  decision  authority  that 


technical  and  operational  risks  have  been  re¬ 
duced  to  acceptable  levels.” 

Enclosure  (2)  Management  Considera¬ 
tions 

Para  3.  “. Acquisition  Time.  Programs 
shall  be  planned  for  system  development 
within  the  shortest  time  reasonable.  At  each 
milestone,  schedule  alternatives  and  inherent 
risks  shall  be  assessed.  Methods  to  be  consid¬ 
ered  include  combination  or  omission  of  acqui¬ 
sition  phases;  smooth  transition  to  production; 
single  concept  development;  preplanned  prod¬ 
uct  improvement;  use  of  alternatives  in  high 
risk  areas;  experimental  prototyping  of  critical 
components;  or  coordination  of  common  pur¬ 
chases  between  different  programs." 

Para  10.  “ Test  and  Evaluation.  Test  and 
evaluation  are  an  integral  part  of  the  acquisi¬ 
tion  process  to  assess  technical  performance 
and  risks, ...  Schedules  shall  be  flexible  to  allow 
retest  or  revaluation  as  necessary  prior  to  a 
milestone,  and  shall  avoid  duplication  com¬ 
mensurate  with  risk.” 

Para  14.  ‘ Acquisition  Risks.  Technical, 
operational,  schedule,  and  cost  risks  shall  be 
identified  as  early  as  possible  and  assessed  con¬ 
tinuously.  They  shall  be  disclosed  in  full  to  the 
decision  authority  and  addressed  realistically 
at  each  milestone.  A  management  reserve 
base  on  the  cost  risk  shall  be  established  for 
ACAT  I  and  IIS  programs.” 

Enclosure  (4)  Navy  Decision  Coordinating  Pa¬ 
per  (NDCP)  Format 

Para  1.  “Risks.  State  program  risk,  in¬ 
cluding  at  Milestone  1.  technological  risks  to  be 
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reduced  by  R&D  and  validated  by  T&E  be¬ 
fore  Milestone  II;  at  Milestone  II,  demon¬ 
strate  that  all  significant  risk  areas  have  been 
resolved  and  verify  that  technology  is  in  hand 
and  only  engineering  (rather  than  experimen¬ 
tal)  effort  remains;  at  Milestone  III,  identify 
any  shortfalls  in  technical  evaluation 
(TECHEVAL)  and  OPEVAL  results  against 
thresholds.” 

Naval  Materiel  Command  Instruction  (NAV- 
MATINST)  5000.29A  Acquisition  Strategy  Pa - 
per  (6  May  1983). 

Para  2.  The  Acquisition  Strategy  Paper 
shall  discuss  Risk  Analysis  in  Section  11  -  Risk 
Analysis  Enclosure  (1)  “specify  the  major 
problems  or  risk  areas  which  have  been  con¬ 
sidered  in  selection  of  an  acquisition  strategy 
and  which  must  be  overcome  to  achieve  the  ba¬ 
sic  program  objectives.” 

Para  3.  Section  III  -  “ Strategy  to  Achieve 
Objectives  and  Implementation  shall  contain 
the  ‘Risk  Management  Plan  for  dealing  with  ar¬ 
eas  (technical,  costs,  schedule,  and  logistics)’, 
and  the  business  management  plan  of  ‘incen¬ 
tives  to  achieve  program  thresholds  including 
methods  to  control  costs’,  and  ‘incentives  to 
improve  reliability  and  reduce  support  costs’.” 
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APPENDIX  D 
ACRONYMS/GLOSSARY 
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ACSN  -  Advanced  Change/Study  Notice 

ACAT  -  Acquisition  Category 

ADILD  -  Document  number  prefix  for  docu¬ 
ments  from  the  Defense  Technical  Information 
Center  and  the  Defense  Logistics  Information 
Exchange  (respectively) 

ADM  -  Advanced  Development  Model 

AFSARC  -  Air  Force  Systems  Acquisition  Re¬ 
view  Council 

AFSC  -  Air  Force  Systems  Command 

ALCM  -  Air  Launched  Cruise  Missile 

AMC  -  Army  Materiel  Command  (Army) 

AMT  -  Amalgamated  Military  Improvement 
Plan/Technical  Improvement  Plan  (Navy) 

ARB  -  Acquisition  Review  Board 

ASARC  -  Army  Systems  Acquisition  Review 
Council 

BCE  -  Baseline  Cost  Estimate 

BIT  -  Built-In  Test 

BITE  -  Built-In  Test  Equipment 

Cjr  -  Consequence  of  Failure 

CA1G  -  Cost  Analysis  Improvement  Group 

CCB  -  Configuration  Control  Board 

CDF  -  Cumulative  Distribution  Function 

CDR  -  Commander 


CDR  -  Critical  Design  Review 

CDRL  -  Contract  Data  Requirements  List 

CDS  -  Concept  Description  Sheet 

CE  -  Concept  Exploration 

CER  -  Cost  Estimating  Relationship 

Cl  -  Configuration  Item 

CM  -  Configuration  Management 

CPM  -  Cost  Performance  Measurement 

CPM  -  Critical  Path  Method 

CPR  -  Cost  Performance  Report 

CRISD  -  Computer  Resources  Integrated  Sup¬ 
port  Document 

CRLCMP  -  Computer  Resources  Life  Cycle 
Management  Plan 

CSC  -  Computer  Software  Component 

CSCI  -  Computer  Software  Configuration 
Item 

C/SCS  -  Cost/Schedule  Control  System 

C/SCSC  -  Cost/Schedule  Control  System  Cri¬ 
teria 

CSOM  -  Computer  Systems  Operator’s  Man¬ 
ual 

CSSR  -  Cost  Schedule  Status  Report 
CWBS  -  Contract  Work  Breakdown  Structure 
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DA  -  Department  of  the  Army 

DAB  -  Defense  Acquisition  Board 

DARCOM  -  U.S.  Army  Development  and 
Readiness  Command 

DCP  -  Decision  Coordinating  Paper 

DCSRDA  -  Deputy  Chief  of  Staff  for  Research, 
Development,  and  Acquisition 

DID  -  Data  Item  Description 

DoD  -  Department  of  Defense 

DOE  -  Department  of  Energy 

DOT&E  -  Director  Operational  Test  and 
Evaluation 

DPESO  -  Defense  Product  Engineering  Serv¬ 
ices  Office 

DRA  -  Decision  Risk  Analysis 
DS  -  Design  Sheet 

DSARC  -  Defense  Systems  Acquisition  Re¬ 
view  Council  (Changed  to  JRMB  and  then  to 
DAB) 

DSMC  -  Defense  Systems  Management  Col¬ 
lege 

DSSP  -  Defense  Standardization  and  Specifi¬ 
cation  Program 

DT  -  Development  Test 

DTC  -  Design  to  Cost 

DT&E  -  Development  Test  and  Evaluation 


DUSDRE  -  Deputy  Under  Secretary  of  De¬ 
fense  for  Research  and  Engineering 

DIV  -  Demonstration/Validation 

ECP  -  Engineering  Change  Proposal 

ECR  -  Embedded  Computer  Resources,  Engi¬ 
neering  Change  Request 

EDM  -  Engineering  Development  Model 

EDT  -  Engineering  Development  Test 

EIMS  -  End  Item  Maintenance  Sheet 

EMV  -  Estimated  Monetary  Value 

FCA  -  Functional  Configuration  Audit 

FFBD  -  Functional  Flow  Block  Diagram 

FIS  -  Facility  Interface  Sheet 

FM  -  Field  Manual  (Army) 

FQR  -  Formal  Qualification  Review 

FSD  -  Full  Scale  Development 

GFE  -  Government-Furnished  Equipment 

GFP  -  Government-Furnished  Property 

HQDA  -  Headquarters,  Department  of  the 
Army 

HWCI  -  Hardware  Configuration  Item 
ICA  -  Independent  Cost  Analysis 
ICD  -  Interface  Control  Document 
ICE  -  Independent  Cost  Estimate 
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ICWG  -  Interface  Control  Working  Group 

ILS  -  Integrated  Logistic  Support 

ILSP  -  Integrated  Logistic  Support  Plan 

IMIP  -  Industrial  Modernization  Incentives 
Program 

IOC  -  Initial  Operating  Capability 

IPR  -  In  Process  Review 

IPS  -  Integrated  Program  Summary 

IRA  -  Industrial  Resource  Analysis 

JMSNS  -  Justification  for  Major  System  New 
Start 

JRMB  -  Joint  Requirements  and  Management 
Board  (Formerly  DSARC,  now  DAB) 

LCC  -  Life  Cycle  Cost 

LCCP  -  Life  Cycle  Cost  Plan 

LLCSC  -  Lower-Level  Computer  Software 
Components 

LOI  -  Letter  of  Instruction 

LRIP  -  Low  Rate  Initial  Production 

LRU  -  Line  Replaceable  Unit 

LSA  -  Logistic  Support  Analysis 

LSAR  -  Logistic  Support  Analysis  Record 

MCCR  -  Mission-Critical  Computer  Re¬ 
sources 

MCCS  -  Mission-Critical  Computer  System 


MICOM  -  U.S.  Army  Missile  Command 

MIP  -  Military  Improvement  Plan  (Navy) 

MM/CC  -  Milestone  Measurement/  Cost 
Correlation 

MR  -  Management  Reserve 

MR  -  Material  Readiness 

MTBF  -  Mean  Time  Between  Failure 

MTBM  -  Mean  Time  Between  Maintenance 

MTBO  -  Mean  Time  Between  Overhaul 

MTTR  -  Mean  Time  To  Repair 

NATO  -  North  Atlantic  Treaty  Organization 

NAVAIR  -  Naval  Air  Systems  Command 

NAVSEA  -  Naval  Sea  Systems  Command 

OCD  -  Operational  Concept  Document 

OFPP-  Office  of  Federal  Procurement  Policy 

O&M  -  Operation  and  Maintenance 

OMB  -  Office  of  Management  and  Budget 

ONAS  -  Office  of  Naval  Acquisition  Support 

OPERA  -  Open  Plan  Extension  for  Risk 
Analysis 

OPEVAL  -  Operational  Evaluation 

O&S  -  Operating  and  Support,  Operation 
and  Support 

OSD  -  Office  of  the  Secretary  of  Defense 
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OT  -  Operational  Test 

OTA  -  Operational  Test  Agency 

OT&E  -  Operational  Test  and  Evaluation 

Pf  -  Probability  of  Failure 

P3/  -  Pre-Planned  Product  Improvement 

PAE  -  Physical  Achievement  Event 

PCA  -  Physical  Configuration  Audit 

PCE  -  Program  Office  Cost  Estimates 

PDF  -  Probability  Density  Function 

PDM  -  Program  Decision  Memorandum 

PDR  -  Preliminary  Design  Review 

PEA  -  Probabilistic  Event  Analysis 

PEP-  Producibilitv  Engineering  and  Planning 

PERT  -  Program  Evaluation  Review  Tech¬ 
nique 

PI  -  Product  Improvement 

PIP  -  Product  Improvement  Plan  (Army) 

PIRN  -  Preliminary  Interface  Revision  Notice 

PM  -  Program  Manager 

PMD  -  Program  Management  Directive 

PMF  -  Probability  Mass  Function 

PMI  -  Proposed  Military  Improvement  (Navy) 

PMP  -  Program  Management  Plan 

POM  -  Program  Objectives  Memorandum 


PROSIM  -  Program,  Project,  or  Process  Simu¬ 
lator 

PRR  -  Production  Readiness  Review 

PS  -  Production  Sheet 

PWBS  -  Program  Work  Breakdown  Structure 

RCE  -  Risk  Cost  Estimate 

MD  -  Research  and  Development 

RDT&E  -  Research,  Development,  Test  and 
Evaluation 

RFM  -  Requiring  Financial  Manager 

RFP  -  Request  for  Proposal 

RISNET  -  Risk  Information  Systems  and  Net¬ 
work  Evaluation  Technique 

R&M  -  Reliability  and  Maintainability 

R/M/A  -  Reliability/Maintainability/ 
Availability 

ROIRO  -  Roll  On/Roll  Off 

SBD  -  Schematic  Block  Diagram 

SCN  -  Specification  Change  Notice 

SCP  -  System  Concept  Paper 

SDDM  -  Secretary  of  Defense  Decision 
Memorandum 

SDR  -  System  Design  Review 
SECDEF  -  Secretary  of  Defense 
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SEMP  -  System  Engineering  Management 
Plan 

SOW-  Statement  of  Work 

SQEP  -  Software  Quality  Evaluation  Plan 

SRR  -  System  Requirements  Review 

SRS  -  Software  Requirements  Specification 

SSA  -  Source  Selection  Authority 

SSAC  -  Source  Selection  Advisoiy  Council 

SSARC  -  Service  System  Acquisition  Review 
Council 

SSR  -  Software  Specification  Review 

STOG  -  Science  and  Technology  Objectives 
Guide 

TDRS  -  Tracking  and  Data  Relay  Satellite 
T&E  -  Test  and  Evaluation 
TECHEVAL  -  Technical  Evaluation 
TEMP  -  Test  and  Evaluation  Master  Plan 


TIP  -  Technical  Improvement  Plan  ' 

TLCSC  -  Top-Level  Computer  Software 
Component 

TLS  -  Time  Line  Sheet 

TPM  -  Technical  Performance  Measurement 

TRACE  -  Total  Risk  Assessing  Cost  Estimate 

TRACE-P  -  Total  Risk  Assessing  Cost 
Estimate  for  Production 

TRR  -  Test  Requirements  Review 

TRS  -  Test  Requirements  Sheet 

TSR  -  Trade  Study  Report 

USA  -  U.S.  Army 

USAF-  U.S.  Air  Force 

USN-  U.S.  Navy 

VERT  -  Venture  Evaluation  Review  Tech¬ 
nique 

WBS  -  Work  Breakdown  Structure 
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Acquisition  Environment  -  The  totality  of  poli¬ 
cies,  practices  and  practical  considerations 
relative  to  management  of  acquisition  pro¬ 
grams. 

Acquisition  Plan  -  Encompasses  program  ob¬ 
jectives,  direction,  and  control  through  the  in¬ 
tegration  of  strategic,  technical,  and  resource 
concerns.  Ideally,  the  acquisition  strategy  is 
structured  at  the  outset  of  the  program  to  pro¬ 
vide  an  organized  and  consistent  approach  to 
meeting  program  objectives  within  known  con¬ 
straints. 

Activity  -  A  program  element  consuming  time 
and  resources.  It  can  be  zero  if  it  is  a  constraint. 

Arc  -  The  line  connecting  two  points  in  a  net¬ 
work. 

Coefficient  of  Variation  -  Ratio  of  standard  de¬ 
viation  to  expected  value.  (See  Standard  De¬ 
viation  and  Expected  Value).  A  measure  of 
relative  uncertainty. 

Confidence  Interval  -  Limits  of  an  uncertain 
quantity  (like  cost)  between  which  there  is  a 
given  probability  of  occurrence.  Expressed  as 
in  “the  n  percent  confidence  interval”.  The 
confidence  level  is  the  left  hand  lower  confi¬ 
dence  interval,  so  that  one  may  say,  “C  is  the 
nth  confidence  level”,  meaning  there  is  an  n 
percent  probability  of  cost  being  between  O 
and  C. 

Confidence  Level  -  Percentile. 

Consistent  Judgment  Matrix  -  A  judgment  ma¬ 
trix  that  expresses  relationships  like  probabili¬ 
ties,  so  that  if  probability  of  /  is  m  times  that  of 


/,  and  J  is  n  times  that  of  K,  then  the  probability 
of  /  is  mn  times  that  of  K.  Since  each  entry  is  a 
ratio,  ry ,  of  the  probability  of  /  divided  by  the 
probability  of/,  then  r^-  times  r ^  equals  r-^.. 

Constraint  -  An  activity  that  does  not  consume 
time  or  resources.  It  acts  as  a  connector  be¬ 
tween  milestones  or  events. 

CER  -  Cost  Estimating  Relationship.  An  esti¬ 
mating  relationship  in  which  cost  of  a  system  is 
the  mathematical  result  of  a  formula  having  se¬ 
lected  system  measurements  (like  thrust  or 
weight)  as  values  in  the  formula. 

Cost  Risk  -  The  risk  to  a  program  in  terms  of 
overrunning  the  program  cost. 

Critical  Index  -The  number  of  times  each  ac¬ 
tivity  appears  on  the  critical  path  during  simu¬ 
lation. 

Critical  Path  -  A  path  with  no  slack  or  float. 

CPM  -  Critical  Path  Method  similar  to  PERT 
but  activity  oriented  with  single  time  estimates. 

Cumulative  Distribution  Function  -  A  curve  or 
mathematical  expression  which  associates  a 
probability  to  all  values  in  the  set  of  values 
over  which  it  is  defined,  so  that  the  probability 
is  that  of  the  occurrence  of  a  value  less  than  or 
equal  to  a  given  value. 

Decision  Analysis  -  Examination  of  decision 
problems  by  analysis  of  the  outcomes  of  deci¬ 
sion  alternatives,  the  probabilities  of  arrival  at 
those  outcomes,  and  the  intervening  decisions 
between  selection  of  alternatives  and  arrival  of 
outcomes.  The  attributes  of  the  outcomes  are 


examined  and  numerically  matched  against 
preference  criteria. 

Decision  free  -  Representation  of  a  decision 
problem  as  paths  from  a  present  decision 
through  alternative,  intermediate  decisions 
and  risky  events  to  outcomes.  The  representa¬ 
tion  is  similar  to  an  increasingly 
branched  tree. 

Deterministic  -  A  term  generally  used  to  refer 
to  a  single  iteration  of  a  risk  network  that  has 
constants  reflecting  “most  likely”  values  as  in¬ 
put  parameters.  As  opposed  to  “Probabalistic” 
which  has  distributions  as  input  parameters 
that  may  be  sampled  many  times. 

Delphi  Technique  -  The  use  of  a  group  of 
knowledgeable  individuals  to  arrive  at  an  esti¬ 
mate  cf  an  uncertain  situation. 

Programmatic  Method  -  A  way  of  describing 
an  expert’s  uncertainty  by  presenting  a  range 
of  PDF  diagrams  with  a  selected  general  shape. 

Engineering  Change  Order  Allowance  -  A 
budget  category  to  be  used  for  funding  changes 
in  the  physical  or  performance  characteristics 
of  a  system. 

Expected  Value  -  The  probabilistic  average  of 
an  uncertain  quantity.  It  equals  the  sum  of  all 
the  products  of  each  considered  value  times  its 
corresponding  probability.  Also  called  the 
mean  when  applied  to  all  possible  values  of  the 
uncertain  quantity. 


Gantt  Chart  -  A  bar  graph  of  horizontal  bars 
showing  program  element  commencement  and 
completion  against  time. 

Histogram  -  A  vertical  bar  chart.  A  method 
often  used  to  represent  a  Probability  Mass 
Function  (PMF). 

Incentive  Share  Ratio  -  The  ratio  of  govern- 
ment-to-contractor  assumption  of  cost  or  sav¬ 
ings  related  to  contract  target  cost. 

Independence  (also  statistical  independence)  - 
The  relationship  between  two  or  more  events 
when  knowledge  of  the  probability  of  occur¬ 
rence  of  one  does  not  alter  the  probability  of 
another. 

ILSP  -  Integrated  Logistics  Support  Plan:  The 
plan  that  defines  the  methods  to  be  used  in 
supporting  the  system  once  it  is  deployed. 

Judgment  Matrix  -  A  square  array  of  values 
such  that  all  entries  are  positive  for  every  en¬ 
try  in  row  i  and  column  j  there  is  an  entry  in 
rowj  and  column  i  which  is  the  reciprocal  of 
the  first. 

Life  Cycle  Cost  ( LCQ :  An  approach  to  costing 
that  considers  all  costs  (Government  and  Con¬ 
tractors)  incurred  during  the  projected  life  of 
the  system,  subsystem,  or  component.  It  in¬ 
cludes  total  cost  of  ownership  over  the  system 
life  cycle  including  all  research,  development, 
test  and  evaluation,  initial  investment,  produc- 
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tion,  and  operating  and  support  (maintenance) 
cost. 

Management  Reserve  -  An  amount  of  budget 
held  aside  from  direct  allocation  to  program 
elements  as  a  reserve  for  contingencies. 

Manufacturing  Plan  -  The  plan  that  contains 
the  details  of  how  the  system  is  to  be  manufac¬ 
tured.  Includes  the  make  or  buy  list  of  the 
equipment. 

Mode  -  A  point  on  a  probability  density  func¬ 
tion  wuere  the  probability  goes  from  increas¬ 
ing  to  decreasing,  that  is,  a  maximum. 

Model  -  A  partial  description  of  a  system  using 
sufficient  detail  for  some  analytic  or  descriptive 
purpose. 

Modified  Churchman  -  Ackoff  Method  -  A 
means  of  ordering  events  in  terms  of  likelihood 
to  occur. 

Moment  -  A  function  (called  the  expectation) 
of  a  probability  law,  often  referred  to  as  an  “nth 
moment”,  where  n  is  any  number  and  denotes 
an  exponent  on  the  uncertain  quantity.  For  ex¬ 
ample,  if  oc  is  a  discrete  uncertain  quantity,  the 
third  moment  is  the  sum  of  all  values  of*3  times 
the  probability  of  each  respective  value  of  x. 

Monte  Carlo  -  The  simulation  technique  in 
which  outcomes  of  events  are  determined  by 
selecting  random  numbers  subject  to  a  defined 
probability  law.  If  the  random  number  falls 
within  the  limits  of  an  outcome’s  probability, 
that  outcome  is  chosen. 


Multiplicative  Cost  Elements  -  Cost  elements 
whose  value  is  derived  by  a  multiplication  of 
other  cost  elements. 

Network  -  A  collection  of  points  connected  by 
lines. 

Network  Based  Schedule  -  An  objective  ori¬ 
ented  plan  of  action  that  includes  all  important 
activities  and  events. 

Network  Program  Model  -  Representation  of  a 
program  by  means  of  a  network  in  which  the 
points  (nodes)  stand  for  program  decision 
points  or  milestones  and  the  lines  (arcs)  stand 
for  program  activities  which  extend  over  time 
and  consume  resources.  Nodes  may  be  re¬ 
garded  as  activities  requiring  no  time  to  com¬ 
plete. 

Node  -  One  of  a  collection  of  points  defining  a 
network. 

Normalized  Geometric  Mean  Vector  Method  -  A 
technique  devised  to  determine  the  assignment 
of  individual  event  probabilities  and  fulfill  the 
axioms  of  probabilities. 

Objective  Probability  -  Probability  which  can  be 
inferred  from  objective  facts. 

Odds  -  The  ratio  of  probabilities  of  occurrence 
and  non-occurrence;  e.g.,  for  a  throw  of  a  fair 
die  the  probability  of  a  four  is  1  / 6.  The  odds  are 
5  to  1. 

Parametric  Cost  Estimating  -  Cost  estimating 
by  means  of  obtaining  information  from  a  data 
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bank  by  specific  parameters  such  as  weight, 
size,  material  composition,  etc. 

Path  -  A  sequence  of  arcs. 

Percentile  -  The  value  of  an  uncertain  quantity, 
generally  referred  to  as  an  “nth  percentile”, 
which  is  greater  than  or  equal  to  n  percent  of  all 
values. 

PERT  -  Program  Evaluation  and  Peview  Tech¬ 
nique.  An  early  network  analysis  technique  for 
acquisition  programs,  in  which  each  activity  du¬ 
ration  was  characterized  by  its  mean  or  ex¬ 
pected  values  and  no  uncertainties  were 
incorporated. 

Probabilistic  Event  Analysis  -  Risk  assessment, 
using  a  variation  of  the  decision  analysis 
method,  developed  in  reference  [54]  of  Appen¬ 
dix  B,  Bibliography,  Basic  Discussion. 

Probability  Density  Function  (PDF)  -  A  prob¬ 
ability  expression  such  that  the  area  under  the 
function  between  defined  limits  of  the  values 
on  which  it  is  defined  represents  the  probabil¬ 
ity  of  the  values  within  those  limits. 

Probability  Function  -  A  mathematical  expres¬ 
sion,  defined  for  an  uncertain  quantity,  associ¬ 
ating  a  probability  with  each  value  or 
non-redundant  combination  of  values  in  the 
set. 

Probability  Mass  Function  (PMF)  -  A  function 
assigning  probabilities  to  each  value  of  uncer¬ 
tain  quantity  having  only  discrete  or  discon¬ 
tinuous  values. 


Program  Advocacy  -  The  personal  interest  in 
the  program  under  study  to  the  exclusion  of 
other  programs  usually  without  merit. 

Program  Management  Directive  (PMD)  -  A 
document  containing  the  goals  of  the  program, 
usually  set  up  as  requirements  such  as  cruising 
speed,  dash  capability,  etc. 

Program  Management  Plan  (PMP)  -  The  pro¬ 
gram  plan  from  feasibility  to  phase  out  of  the 
system. 

Program  Risk  -  The  probability  of  not  achiev¬ 
ing  a  defined  cost,  schedule,  or  technical  per¬ 
formance  goal. 

Programmatic  Risk  -  The  risks  involved  in  ob¬ 
taining  and  using  applicable  resources  and  ac¬ 
tivities  that  are  outside  of  the  programs 
control,  but  can  affect  the  program’s  direction. 

Random  Number  Generator  -  A  computer  pro¬ 
gram  capable  of  providing  numbers  able  to 
pass  statistical  tests  indicating  that  any  number 
between  the  limits  of  those  generated  is  equally 
as  likely  to  be  generated. 

Regression  Analysis  -  Determination  of  the  val¬ 
ues  of  constants  in  a  mathematical  expression 
which  gives  results  that  are  the  closest  to  the 
observed  values  associated  with  values  of  the 
data  used  in  the  expression.  For  example,  if 
cost  C  is  assumed  to  be  the  sum  of  a  fixed  cost, 
F,  and  variable  cost,  V,  for  N  items,  C = F+  VN. 
If  data  shows  the  expression  tc.  be  inexact,  re¬ 
gression  analysis  finds  vaiuesof  Fand  1/ which 
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give  the  value,  C,  closest  to  those  associated 
with  all  data  values  of  N.  Regression  Analysis  is 
a  process  by  which  the  relationship  between 
paired  variables  can  be  described  mathemati¬ 
cally  using  the  tendency  of  jointly  correlated 
random  variables  to  approach  their  mean. 

Risk  -  The  condition  of  having  outcomes  with 
known  probabilities  of  occurrence,  not  cer¬ 
tainty  of  occurrence. 

Risk  -  The  combination  of  the  probability  of  an 
event  occurring  and  the  significance  of  the  con¬ 
sequence  of  the  event  occurring. 

Risk  Analysis  -  Involves  an  examination  of  the 
change  in  consequences  caused  by  changes  in 
the  risk-input  variables. 

Risk  Assessment  -  The  process  of  examining  ail 
aspects  of  a  program  with  the  goal  of  identify¬ 
ing  areas  of  risk  and  the  corresponding  poten¬ 
tial  impact. 

Risk  Assumption  -  A  conscious  decision  to  ac¬ 
cept  the  consequences  of  the  risk  occurring. 

Risk  Avoidance  -  Risk  avoidance  is  to  non-se¬ 
lect  an  option  because  of  potentially  unfavor¬ 
able  results.  Selection  of  an  option  because  of 
lower  risk  is  also  risk  avoidance. 

Risk  Control  -  Risk  control  is  the  process  of 
continually  monitoring  and  correcting  the 
condition  of  the  program. 

Risk  Drivers  -  The  technical,  programmatic, 
and  supportability  risk  facets. 


Risk  Handling  -  The  last  critical  element  in  the 
risk  management  process.  It  is  the  action  or  in¬ 
action  taken  to  address  the  risk  issues  identi¬ 
fied  and  evaluated  in  the  risk  assessment  and 
risk  analysis  efforts. 

Risk  Identification  -  Narrative  statements  de¬ 
scribing  the  program  risks. 

Risk  Management  -  Relates  to  the  various  proc¬ 
esses  used  to  manage  risk. 

Risk  Planning  -  Forcing  organized  purposeful 
thought  to  the  subject  of  eliminating,  mini¬ 
mizing,  or  containing  the  effects  of  undesirable 
occurrences.  It  allows  for  (1)  eliminating  risk 
wherever  possible;  (2)  isolating  and  minimiz¬ 
ing  risk;  (3)  developing  alternate  courses  of 
action;  and,  (4)  establishing  time  and  money  re¬ 
serves  to  cover  risks  that  can  be  avoided. 

Risk  Rating  Scheme  -  A  method  of  assigning  a 
risk  level  such  as  high,  medium  ,  or  low  risk 
based  on  an  agreed  value  assigned  to  the  prob¬ 
ability  of  occurrence  and  the  severity  of  the  im¬ 
pact  of  failure. 

Risk  Transfer  -  The  sharing  of  risk  through 
contractual  agreements  such  as  performance 
incentives,  warranties,  etc.  It  can  also  be  be¬ 
tween  government  agencies  as  in  multi-service 
programs. 

Schedule  Risk  -  The  risk  to  a  program  in  not 
meeting  the  major  milestones. 
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Simulation  -  The  operation  of  a  model  which 
provides  outputs  analogous  to  the  system  mod¬ 
eled. 

Skew  -  The  asymmetry  of  a  probability  density 
function.  The  skew  is  to  the  side  of  the  mode 
under  which  lies  the  greatest  area . 

Skewness  -  The  measure  of  the  amount  of 
skew. 

Slack  -  The  difference  between  the  earliest 
possible  completion  time  of  a  path  or  activity 
and  its  latest  possible  completion  time. 

Standard  Deviation  -  The  square  root  of  the 
variance.  Often  used  because  it  is  in  the  same 
units  as  the  random  variable  itself,  and  can  be 
depicted  on  the  same  axes  as  the  Probability 
Density  Function  of  which  it  is  a  characteristic. 

Standard  Normal  Function  -  A  probability 
function  centered  on  zero,  with  a  standard  de¬ 
viation  of  1,  having  a  bell  shape  and  covering 
values  that  become  negatively  and  positively 
infinite. 

Subjective  Probability  -  An  expression  of  pre¬ 
dictability  in  terms  of  personal  statements 
obeying  the  axioms  of  probability  and  equal  to 
the  probabilities  acceptable  to  the  assessor  for 
a  substitute  gamble. 

Supportability  Risk  -  The  risks  associated  with 
fielding  and  maintaining  systems  which  are  cur¬ 
rently  being  developed  or  have  been  developed 
and  are  being  deployed. 


SEMP  -  Systems  Engineering  Management 
Plan.  The  plan  for  the  system  engineering  as¬ 
pects  of  a  program. 

Technical  Risk  -  The  risk  associated  with  evolv¬ 
ing  a  new  design  to  provide  a  greater  level  of 
performance  than  previously  demonstrated. 
Includes  the  same  or  lesser  level  of  perform¬ 
ance  subject  to  new  constfaints  such  as  size  or 
weights. 

TEMP  -  Test  and  Evaluation  Master  Plan.  The 
plan  for  all  required  testing  and  evaluation  of  a 
system. 

Uncertainty  -  The  condition  of  having  out¬ 
comes  with  unknown  probabilities  of  occur¬ 
rence. 

Uniform  Distribution  -  A  set  of  values  where 
every  value  has  an  equal  probability  of  occur¬ 
rence. 

Utility  Theory  -  Theory  of  preference  under 
conditions  of  risk. 

Variance  -  A  measure  of  the  variability  of  a  ran¬ 
dom  variable.  The  standard  deviation 
squared.  Often  symbolized  as  Var  (  ). 

Work  Breakdown  Structure  -  A  product  ori¬ 
ented  family  tree  division  of  hardware,  soft¬ 
ware,  services,  and  other  work  tasks  which 
organizes,  defines,  and  graphically  displays  the 
product  to  be  produced,  as  well  as  the  work  to 
be  accomplished  to  achieve  the  specified  prod¬ 
uct. 
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APPENDIX  E 

BASIC  PROBABILITY  CONCEPTS 
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1.  INTRODUCTION 

This  appendix  serves  as  a  very  basic  in¬ 
troduction  to  probability  and  statistical  con¬ 
cepts  that  may  be  useful  for  risk  analysis.  The 
appendix  is  by  no  means  all  inclusive  but  rather 
may  be  thought  of  as  a  primer.  The  appendix  is 
divided  into  three  sections.  The  first  section  is 
an  introduction  to  probability  centering  on 
definitions  and  simple  examples.  The  second 
section  begins  with  a  summary  of  descriptive 
statistics  including  a  look  at  statistical  confi¬ 
dence  and  confidence  intervals.  The  second 
section  then  gives  an  explanation  of  probabil¬ 
ity  density  functions  (PDFs)  and  cumulative 
density  functions  (CDFs),  which  define  distri¬ 
butions  such  as  the  Uniform,  Normal,  and  Tri¬ 
angular  which  are  relevant  to  risk  analysis. 
The  third  section  discusses  statistical  inde¬ 
pendence,  which  is  the  prerequisite  for  the  con¬ 
cept  of  expected  values.  Decision  tree  analysis 
is  illustrated  to  show  the  merit  of  the  expected 
values  approach. 

Probability 

Probability  is  a  concept  used  by  many 
people  everyday.  As  an  example  the  weather¬ 
man  predicts  a  30  percent  probability  of  rain. 
This  means  that  in  the  long  run  one  might  ex¬ 
pect  rain  30  days  out  of  100  when  conditions 
are  the  same  as  they  are  at  the  time  the  forecast 
is  made.  For  risk  analysis  a  statement  might  be 
made  to  the  effect  that  the  developmental  st.'  ge 
of  weapon  system  A  has  a  10  percent  probabil¬ 
ity  of  a  schedule  (time)  overrun.  This  is  equiva¬ 
lent  to  saying  that  of  all  the  developmental 


stages  of  weapon  systems  similar  to  A,  in  the 
past  10  percent  have  had  a  schedule  overrun. 

More  formal  definitions  of  probability 
are  given  below. 

PROBABILITY  -  “1.  The  quality  or 
condition  of  being  probable;  likelihood.  2.  A 
probable  situation,  condition,  or  event.  3. 
Math.  A  number  expressing  the  likelihood  of 
occurrence  of  a  specific  event,  such  as  the  ratio 
of  the  number  of  experimental  results  that 
would  produce  the  event  to  the  total  number  of 
results  considered  possible.”  ( The  American 
Heritage  Dictionary ). 

PROBABILITY  -  “In  practical  situ¬ 
ations,  probability  is  used  as  a  vehicle  in  draw¬ 
ing  inferences  about  unknown  population 
characteristics.  Additionally,  ...,  probability 
concepts  can  be  used  to  give  us  an  indication  of 
how  good  these  inferences  are,"  ( Statistical 
Methods  for  Business  and  Economics)  Pffen* 
berger  and  Patterson,  1977.  Reference  1. 

Many  individuals  think  of  probability 
in  relation  to  gambling  and  games  of  chance 
such  as  card  playing  and  dice  throwing.  They 
measure  the  probability  of  an  event  in  terms  of 
the  odds  against  the  event  happening.  One  fur¬ 
ther  example,  the  throwing  of  a  pair  of  dice,  will 
illustrate  the  inverse  relationship  between 
probability  and  “the  odds  against  an  event.” 
Throwing  an  ordinary  pair  of  dice  results  in  one 
of  thirty-six  possible  outcomes.  These  are  il¬ 
lustrated  by  Figure  E-l. 

The  probability  of  throwing  a  “10"  is 
3/36  or  0.083.  This  is  three  out  of  the  thirty-six 
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possible  outcomes  result  in  a  “10”.  The  odds 
against  throwing  a  “10”  are  “11  to  1.”  This  is 
since  the  total  number  of  possible  non-10  out¬ 
comes,  thirty-three,  is  eleven  times  the  num¬ 
ber  of  outcomes,  three,  which  result  in  a  “10”. 

Probability  is  a  key  quantitative  meas¬ 
ure  associated  with  many  risk  assessment  tech¬ 
niques.  The  above  examples  are  simplistic  but 
show  how  easy  it  is  to  comprehend  probability 
concepts.  The  next  two  sections  expand  on  the 
basic  premise  of  probability  understanding. 

Descriptive  Statistics,  Confidence, 
and  Distributions 

Any  group  of  numbers,  such  as  a  sample 
composed  of  quantitative  evaluations,  may  be 
described  with  the  following  basic  statistical 
parameters: 

•  mean  •  mode 

•  median  •  variance  and 

•  range  standard  deviation 

These  parameters  enable  the  statisti¬ 
cian  to  determine  what  level  of  confidence  (or 
assurance)  may  be  accorded  to  predictive  state¬ 
ments  about  the  entire  population  of  numbers. 
The  parameters  also  help  determine  of  what 
possible  statistical  distribution  the  sample  is  a 
part.  Conversely,  a  statistical  distribution  may 
be  described  by  such  parameters.  A  statistical 
distribution  is  basically  just  a  way  to  describe 
which  numbers  will  appear  more  often  (or  with 
a  high  probability),  and  which  numbers  will 
appear  less  often  (or  with  a  low  probability). 
The  following  paragraphs  define  the  parame¬ 
ters  in  some  detail  and  then  discuss  confidence 


levels,  PDFs  and  CDFs,  and  the  other  relevant 
distributions  applied  in  risk  analysis. 

For  illustrative  purposes  let  the  follow¬ 
ing  numbers  represent  exam  scores  for  a  ficti¬ 
tious  introductory  statistics  course: 


75 

60 

100 

65 

80 

45 

25 

45 

60 

90 

60 

40 

50 

70 

55 

10 

95 

70 

85 

20 

70 

65 

90 

90 

65 

70 

80 

70 

55 

Let*  represent  these  numbers,  where 
i  is  indexed  from  1  to  29.  So  XI  =  75,  X2  =  80, 

X3  «  60 . X28  =  90, X29  =  55.  The  mean  of 

these  numbers  is  nothing  more  than  the  arith¬ 
metic  average.  The  mean  is  computed  thus: 

n 

Y,_Xi 

MEAN  =  =  63.96 

n  29 

where  ,i  is  the  number  of  exam  scores.  The 
mode,  the  most  likely  or  probable  score,  is  70. 
The  mode  occurred  five  times  more  often  than 
any  other  score.  The  median  is  the  middle  score 
if  the  scores  are  ranked  top  to  bottom.  Since 
there  are  twenty-nine  scores  altogether,  the 
median  is  the  fifteenth  score,  which  is  a  65.  The 
variance  and  standard  deviation  of  a  group  of 
numbers  are  an  attempt  to  describe  the  disper¬ 
sion  or  scattering  of  the  numbers  around  the 
mean.  The  variance  is  computed  using  the  fol¬ 
lowing  formula: 
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VARIANCE  = 


n-1 


mean  that  95  percent  of  all  the  possible  values 
that  are  sampled  will  fall  between  56  and  72. 
which  is  the  common,  though  faulty,  interpre¬ 
tation  of  the  statements. 


For  this  example  the  variance  is: 


132275  - 


28 


=  486.4 


The  standard  deviation  is  the  square 
root  of  the  variance.  The  standard  deviation 
has  a  more  intuitive  appeal  than  does  the  vari¬ 
ance  since  the  standard  deviation  is  mathe¬ 
matically  the  average  variation  of  a  value  from 
the  mean.  For  this  example  the  standard  devia¬ 
tion  is  V  486.4  =  22.05.  The  range  is  the  high 
score  minus  the  low  score.  For  this  example, 
the  range  is  100-10=90. 

Many  times  when  examining  data  a 
“level  of  confidence”  or  “confidence  interval" 
is  used  to  indicate  what  certainty  or  faith  is  to 
be  put  in  the  sample  that  is  being  taken  as  rep¬ 
resentative  of  the  entire  population.  Far  and 
away  the  most  common  measure  in  the  area  is 
the  confidence  interval  for  the  mean.  A  state¬ 
ment  such  as  follows  is  made  about  a  particular 
sample  mean: 

“The  95  percent  confidence  interval 
for  the  mean  is  56  to  72.” 

This  statement  means  statistically  that 
of  all  the  possible  samples  of  this  size  taken  out 
of  this  population,  95  percent  of  the  samples 
will  have  a  mean  between  56  and  72.  It  does  not 


Confidence  intervals  are  determined  by 
adding  and  subtracting  some  calculated  value 
from  the  mean  of  the  sample.  Usually,  but  not 
always,  this  value  is  based  on  the  standard  de¬ 
viation  of  the  sample.  As  an  example,  if  the 
population  from  which  a  sample  is  taken  is  de¬ 
termined  to  be  normally  distributed,  and  we 
have  assumed  this  in  the  previous  statements 
(this  determination  may  be  made  based  on  the 
relative  values  of  the  mean,  variance  and  stan¬ 
dard  deviation,  mode,  median,  range,  and 
other  factors),  then  a  95  percent  confidence  in¬ 
terval  for  the  population  is  calculated  in  this 
manner: 


X  ±  1.96  o 


where  X  is  the  sample  mean  and  o  is  the 
standard  deviation.  A  95  percent  confidence 
interval  for  the  mean  is  calculated  in  this 
manner: 


X  ± 


o 

where  Vn"  is  commonly  referred  to  as  the 
standard  error. 

One  might  ask  how  the  population  is 
determined  to  be  normal  (or  normally  distrib¬ 
uted)  in  the  first  place.  Similar  groups  of  num¬ 
bers  have  similar  relationships  between  their 
respective  parameters.  These  similarities  help 
determine  which  distribution  describes  the  en¬ 
tire  population.  Typical  distributions  for  prob- 
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lems  associated  with  risk  are  the  Normal, 
Uniform,  Triangular,  and  Beta.  Discussion  of 
the  Beta  distribution  is  beyond  the  scope  of  this 
appendix.  If  the  reader  requires  further  infor¬ 
mation  on  the  Beta  distribution,  any  of  several 
statistics  and  operations  research  books  readily 
available  can  supply  the  information. 

For  the  normal  distribution,  68.3  per¬ 
cent  of  all  possible  values  lie  within  one  stan¬ 
dard  deviation  of  the  mean,  95.4  percent  lie 
within  two  standard  deviations,  and  99.7  per¬ 
cent  lie  within  three  standard  deviations.  To 
pictorially  show  the  above,  one  can  look  at  the 
Probability  Density  Function  (PDF).  The  PDF 
gives  the  probability  that  certain  values  will  oc¬ 
cur.  Figure  E-2  below  is  a  PDF  for  the  exam 
scores  example,  assuming  that  the  scores  are 
from  a  normal  distribution. 


The  normal  distribution  is,  by  strict 
definition,  a  continuous  distribution.  However, 
Figure  E-2  implies  that  fractional  exam  scores 
are  possible,  but  of  course  it  is  not  realistic  in 
this  example.  A  discussion  of  the  differences 
between  discrete  and  continuous  distribution  is 


beyond  this  appendix,  and  since  the  example  is 
only  meant  to  be  used  for  illustrative  purposes, 
this  finer  point  of  statistics  will  be  ignored.  Fig¬ 
ure  E-2  also  implies  that  extra  credit  is  given 
since  scores  exceeding  100  are  possible,  and 
this  could  certainly  be  within  the  realm  of  our 
example.  The  most  important  distinction  of  the 
normal  distributions  PDF  is  the  bell  shape  of 
the  curve.  This  is  the  most  definitive  character¬ 
istic  of  any  PDF:  shape. 

The  Cumulative  Density  Function 
(CDF)  is  arithmetically  the  summation  of  the 
PDF.  In  plainer  words,  the  CDF  gives  the 
probability  a  value  (or  any  valueless  than  the 
value)  will  occur.  The  shape  of  the  various  dis¬ 
tributions  CDFs  are  distinctive,  and  the  CDF  is 
merely  another  way  of  illustrating  the  distribu¬ 
tion.  Figure  E-3  is  a  typical  CDF  for  normally 
distributed  values,  in  this  case  the  exam  scores 
example. 

The  uniform  distribution  is  used  to  de¬ 
scribe  a  set  of  values  where  every  value  has  an 
equal  probability  of  occurrence.  Returning 
once  again  to  the  exam  scores  example,  one 
might  hypothesize  that  ali  possible  scores  (1, 2, 
3, ...  98, 99, 100, ...)  have  an  equal  probability  of 
occurrence:  0.01.  The  PDF  for  this  is  illus¬ 
trated  by  Figure  E-4. 

Figure  E-5  illustrates  the  uniform 

CDF, 

The  triangular  distribution  is  often 
used  in  risk  analysis  situations  to  describe  the 
most  optimistic,  most  likely,  and  most  pessi¬ 
mistic  durations  of  some  event  or  activity.  The 
PDF  of  the  triangular  distribution,  illustrated 
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Figure  E-3  CDF  of  a  Normal  Distribution 


EXAM  SCORE 

Figure  E-4  PDF  of  a  Uniform  Distribution 


by  Figure  E-6,  is  not  necessarily  symmetric.  In-  right”  to  reflect  the  possibility  of  very  long  time 

deed,  many  times  the  triangular  distribution  is  durations.  These  long  durations  are  less  likely 

purposely  nonsymmetric  or  “skewed  to  the  to  occur  but  do  happen  occasionally.  In  the  ex- 
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Figure  E-5  CDF  of  a  Uniform  Distribution 
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Figure  E-6  PDF  of  a  Triangular  Distribution 


ample  shown  by  Figure  E-6,  one  notices  that 
eight  days  is  the  most  likely  production  time  for 
a  widget  wing.  Clearly  the  average  is  “to  the 
right”  and  is  very  close  to  9.3  days.  Hence,  the 
triangular  distribution,  when  skewed,  has  a 


mode  and  mean  which  are  clearly  different. 
Contrast  this  to  the  normal  distribution,  where 
the  mode  and  mean  are  the  same  (as  is  the  me¬ 
dian). 
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Independence,  Expected  Value, 
and  Decision  Tree  Analysis 

Statistical  independence  is  an  impor¬ 
tant  concept  upon  which  a  good  deal  of  meth¬ 
odologies  are  based.  For  this  appendix  it  is 
important  to  give  a  brief  definition  before  go¬ 
ing  through  the  basic  principles  of  expected 
value  and  decision  tree  analysis. 

Most  discussions  of  statistical  inde¬ 
pendence  begin  with  a  tutorial  on  conditional 
probability,  sample  space,  and  event  relation¬ 
ships.  Rather  than  discuss  these  concepts,  a 
more  intuitive  (practical)  definition  of  statisti¬ 
cal  independence  is  that  two  events  are  said  to 
be  independent  if  the  occurrence  of  one  is  not 
related  to  the  occurrence  of  the  other.  If  events 
are  occurring  at  random,  then  they  are  inde¬ 
pendent.  If  events  are  not  occurring  at  random, 
then  they  are  not  independent.  A  set  or  group 
of  possible  events  are  said  to  be  mutually  ex¬ 
clusive  and  collectively  exhaustive  if  they  are 
all  independent,  and  the  sum  of  their  probabili¬ 
ties  of  occurrence  is  1.0.  This  is  the  basic  no¬ 
tion  behind  expected  value. 

To  illustrate  the  expected  value  con¬ 
cept,  suppose  that  a  game  of  chance  can  be 
played  for  $1.00.  It  is  a  very  simple  game.  The 
bettor  pays  $1.00  and  has  a  chance  to  win 
$50.00.  The  bettor  may  also  win  $2.00  or  no 
money  at  all.  The  dollar  amounts  and  probabil¬ 
ity  of  winning  are  shown  by  Table  E-l. 


Table  E-l  Expected  Values  Example 


AMOUNT 

VALUE 

PROBABILITY 
OF  WINNING 

EXPECTED 

VALUE 

$50.00 

0.01 

0.50 

2.00 

0.10 

0.20 

0.00 

0.89 

0.00 

TOTAL 

1.00 

$0.70 

The  bettor  would  like  to  know,  before 
actually  paying  his  dollar,  what  the  expected 
winnings  are.  The  expected  value  of  winnings  is 
the  sum  of  the  winning  amounts  multiplied  by 
their  respective  probability  of  occurrence  or 

($50.00)  (0.01)  +  ($2.00)  (0.10)  +  ($0.00) 
(0.89)  =  $0.50  +  $0.20  +  $0.00  =  $0.70. 

Since  the  bettor  can  only  expect  win¬ 
nings  on  the  average  of  seventy  cents  and  pays 
one  dollar  to  play  ine  game,  the  net  payoff  is  a 
negative  thirty  cents. 

One  might  believe  that  most  individu¬ 
als,  when  forced  to  face  this  logic,  would 
choose  not  to  play.  However  this  is  a  very  real¬ 
istic  example  of  gambling  and  risk.  Many  indi¬ 
viduals  would  play  this  game.  They  are  willing 
to  accept  the  risk  of  losing  $1.00  in  order  to 
take  a  chance  at  winning  $50.00.  They  are  risk- 
prone.  The  individual  who  follows  the  basic 
logic  of  this  example  and  does  not  play  is  said 
to  be  risk-averse. 

Expected  value  is  a  notion  prerequisite 
to  the  following  discussion  on  Decision  Tree 
Analysis.  Decision  tree  analysis  attempts  to 
break  down  a  series  of  events  into  smaller,  sim- 
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pier,  and  more  manageable  segments.  Many 
similarities  exist  between  decision  tree  analysis 
and  more  complicated  forms  of  management 
and  risk  analysis,  such  as  the  Program  Evalu¬ 
ation  and  Review  Technique  (PERT)  and  the 
Critical  Path  Method  (CPM).  All  three  forms 
of  analysis  presume  that  a  sequence  of  events 
can  be  broken  down  into  smaller  and  smaller 
segments,  therefore  more  accurately  repre¬ 
senting  reality. 

Decision  tree  analysis  helps  the  analyst 
break  a  problem  down  into  various  sectors  or 
branches  in  order  to  simplify  potential  deci¬ 
sion-making.  As  an  example,  suppose  that  a 
widget  is  being  manufactured  in  the  following 
fashion.  Either  machine  A  or  machine  B  can  be 
used  for  the  first  step  (of  a  two-step  manufac¬ 
turing  process)  with  equal  probability  of  0.5. 
Then  the  second  step  has  machine  C  or  D  proc¬ 


essing  the  widget.  Machine  C  is  used  70  per¬ 
cent  of  the  time  if  the  widget  was  first  processed 
with  machine  A,  and  used  40  percent  of  the 
time  if  the  widget  was  first  processed  with  ma¬ 
chine  B.  Otherwise,  machine  D  is  used  for  the 
second  step.  Decision  tree  analysis  can  be  used 
to  help  compute  the  probability  of  the  widget 
being  produced  by  the  various  combinations  of 
machines  (AC,AD,BC,BD).  Figure  E-7  illus¬ 
trates  the  decision  tree  and  the  expected  prob¬ 
ability  for  each  of  the  four  manufacturing 
process  alternatives. 

Note  that  an  alternative’s  probability  is 
merely  the  product  of  the  individual  processes 
making  up  that  alternative,  since  the  individual 
processes  are  independent  of  each  other.  Note 
also  that  the  sum  of  the  probabilities  for  all  of 
the  four  processing  alternatives  is  1.00. 


PROCESS  1  PRQCESS.2 


ALTERNATIVES 


AC  IS  (.5)  (.7)  -  .35 
AD  IS  (.5)  (.3)  -  .15 
BC  IS  (.5)  (.4)  -.20 
BD  IS  (.5)  (.6)  -  .30 

SUM  OF  PROBABILITIES  - 1 .00 


Decision  Tree 
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APPENDIX  F 

QUANTIFYING  EXPERT  JUDGMENT 


F-l 


I.  GENERAL 


All  risk  assessment  techniques  share  a 
common  need,  and  that  is  the  acquisition  of  ex¬ 
pert  judgment  as  input  to  any  of  the  risk  assess¬ 
ment  models.  Inherent  in  judgment  is  a  degree 
of  uncertainty.  When  acquiring  quantifiable 
expressions  of  judgment,  it  is  imperative  that 
the  axioms  of  probability  not  be  violated: 

1)  The  probabilities  of  all  pos¬ 
sible  events  must  sum  to 
one 

2)  The  probability  of  any 
event  P(A)  must  be  a  num¬ 
ber  greater  than  or  equal  to 
zero  and  less  than  or  equal 
to  one  (0  <L  P(A)  <_  1) 

3)  The  probability  of  joint 
events  is  the  product  of  the 
probability  tnat  one  event 
occurs,  and  the  probability 
that  another  event  occurs 

iven  that  the  first  event 
as  occurred  (P(A)  x 
P(B|A).  Under  these  cir¬ 
cumstances,  the  events  are 
termed  dependent 

4)  When  the  probability  of 
joint  events  occurring  is 
simply  the  product  of  the 
probabilities  of  each  P(A)  x 
P(B),  the  events  are  said  to 
be  independent.  That  is,  the 
two  events  have  nothing  in 
common  or  can  occur  si¬ 
multaneously. 


The  challenge  for  the  analyst  is  to  ob¬ 
tain  expert  judgment  in  the  areas  of  cost, 
schedule  and  technical  performance,  which  is 
qualitative  by  nature.  Next,  he/she  must  con¬ 
vert  it  to  a  quantitative  form,  so  that  the  results 
can  be  depicted  in  the  form  of  a  probability 
density  function  (PDF),  which  serves  as  input 


to  the  various  risk  models  (keep  in  mind  that 
this  is  only  necessary  when  a  quantitative 
model  has  been  selected). 

A  probability  density  function  (PDF)  is 
a  smooth  line  or  curve  such  as  shown  in  Figure 
F- 1.  A  PDF  of  a  random  variable*  is  a  listing  of 
the  various  values  of  *  with  a  corresponding 
probability  associated  with  each  value  of  the 
random  variable  *.  For  our  purposes,  *  would 
be  a  cost,  schedule,  or  performance  value. 
Note  that  the  total  area  under  the  curve  equals 
1. 


Figure  F-l  Probability  Density  Function 


Using  Figure  F-l,  the  random  variable 
*  might  represent  a  hardware  system  cost,  the 
probability  of  the  system  costing  $10,000 
would  be  0.13. 

There  are  a  number  of  methods  which 
can  be  used  to  convert  qualitative  judgment 
into  quantitative  probability  distributions.  The 
remainder  of  this  section  will  focus  on  a  few  of 
the  most  popular,  practical,  and  accurate  tech¬ 
niques  for  doing  so.  The  techniques  discussed 
were  selected  because  they  are  relatively  sim- 
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pie  and  easy  to  master.  This  factor  is  of  para¬ 
mount  importance,  because  in  most  cases  the 
analyst  who  will  be  performing  this  task  will 
have  neither  the  time  or  the  knowledge  of  the 
advanced  probability  concepts  required  to  per¬ 
form  more  complex  techniques.  Those  inter¬ 
ested  in  more  exotic,  complex  techniques  are 
referred  to  Section  V  -  Sources  of  Additional 
Information  -  at  the  end  of  this  appendix. 

The  following  techniques  will  be  dis¬ 
cussed  in  this  appendix: 

1)  Diagrammatic 

2)  Direct 

3)  Betting 

4)  Modified  Churchman/ 

Ackoff  technique 

5)  Delphi  Approach. 

II.  DESCRIPTION  OF  TECHNIQUES 

1)  Diagrammatic 

Many  analysts  prefer  the  diagrammatic 
method  as  a  way  of  capturing  and  representing 
an  expert’s  judgment.  This  method  is  a  simple 
way  of  describing  an  expert’s  uncertainty  by 
presenting  him  with  a  range  of  PDF  diagrams 
and  having  the  expert  select  the  shape  of  the 
PDF  which  is  considered  to  reflect  most  accu¬ 
rately  the  schedule,  cost,  or  technical  parame¬ 
ter  in  question.  Using  this  method,  the  analyst 
can  ascertain  whether  the  PDF  is  symmetric  or 
skewed,  the  degree  of  variability,  etc.  For  ex¬ 
ample,  if  the  expert  feels  that  there  is  a  great 
amount  of  risk  associated  with  completing  an 
activity  within  a  certain  period  of  time,  a  PDF 


skewed  to  the  right  may  be  selected.  Likewise, 
activities  with  little  risk  may  be  skewed  to  the 
left.  If  the  expert  feels  that  each  value  over  a 
given  range  is  equally  likely  to  occur,  a  uni¬ 
form  distribution  may  be  most  appropriate. 
The  analyst  and  the  expert,  working  together, 
can  select  the  PDF  which  most  accurately  re¬ 
flect  the  schedule,  cost,  or  technical  item  under 
question. 

The  diagrammatic  method  of  obtaining 
PDFs  is  applicable  when  the  expert  has  a  sound 
understanding  of  probability  concepts  and  can 
merge  that  understanding  with  his  understand¬ 
ing  of  the  parameters  under  question.  In  this 
way  the  expert  can  accurately  identify  the  ap¬ 
propriate  PDFs. 

2)  Direct 

The  direct  method  is  a  relatively  simple 
technique  which  can  be  used  to  obtain  subjec¬ 
tive  probability  distributions  by  asking  the  ex¬ 
pert  to  assign  probabilities  to  a  given  range  of 
values. 

The  direct  method  of  obtaining  PDFs  is 
applicable,  1)  when  questions  can  be  phrased 
to  the  respondents  in  such  a  way  that  there  is  no 
confusion  likely  to  exist  in  the  respondents 
mind,  and  2)  when  the  results  will  not  violate 
the  axioms  of  probability.  This  method  is  appli¬ 
cable  when  time/resource  constraints  do  not 
allow  for  more  complex,  resource  intensive 
methods. 

The  application  of  the  direct  method  is 
quite  simple.  The  analyst  would  define  a  rele¬ 
vant  range  and  discrete  intervals  for  the  pa- 
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rameter  for  which  the  PDF  is  to  be 
constructed.  For  example,  the  analyst  might 
define  the  relevant  time  duration  for  a  program 
activity  (test  of  a  piece  of  equipment)  to  be  be¬ 
tween  0  and  27  days.  The  analyst  would  then 
break  this  relevant  range  down  into  intervals, 
say  intervals  of  three  days,  the  resulting  formu¬ 
lation  would  look  as  follows: 

0-3  days  16  -  19  days 

4-7  days  20  -  23  days 

8-11  days  24  -  27  days 

12  -  15  days 

Given  these  intervals  over  the  relevant 
range,  the  analyst  would  then  query  the  ex¬ 
pert  to  assign  relative  probabilities  to  each 
range.  From  this,  the  form  of  the  PDF  could  be 
identified.  It  is  imperative  that  the  axioms  of 
probability  not  be  violated. 

Besides  the  application  already  de¬ 
scribed,  the  analyst  could  request  the  expert  to 
provide  a  lowest  possible  value,  a  most  likely 
value,  and  a  highest  possible  value.  The  analyst 
then  makes  an  assumption  about  the  form  of 
the  density  function.  That  is,  is  the  PDF  uni¬ 
form,  normal,  beta,  triangular,  etc? 

3)  Betting 

One  method  of  phrasing  questions  to 
experts  in  order  to  obtain  probabilities  for 
ranges  of  values  (cost/schedule)  states  the 
problem  in  terms  of  betting.  A  form  of  this 
method,  which  was  described  by  Winkler 
(1967),  helps  the  expert  (assessor)  assess  prob¬ 
abilities  of  events  which  are  in  accordance  with 
his  judgment.  The  assumption  with  this  method 
is  that  the  judgment  of  the  expert  may  be  fully 


represented  by  a  probability  distribution,  f(x) 
of  a  random  variable*.  This  method  offers  the 
expert  a  series  of  bets. 

Under  ideal  circumstances,  the  bets  are 
actual,  not  hypothetical.  That  is,  in  each  case 
the  winner  of  the  bet  is  determined  and  the 
amount  of  money  involved  actually  changes 
hands.  However,  under  our  circumstances,  this 
is  not  feasible  (or  legal!).  In  each  case,  the  ex¬ 
pert  must  choose  between  two  bets  (the  expert 
is  not  allowed  to  refrain  from  betting).  The  ex¬ 
pert  must  choose  between  a  bet  with  a  fixed 
probability  q  of  winning  and  1  ~q  of  losing,  and 
a  bet  dependent  on  whether  or  not  some  event 
E  (a  particular  program  activity  duration  range, 
or  cost  range)  occurs.  The  bet  can  be  depicted 
as  follows: 

Bet  la  -  win  $A  if  the  event  E  occurs 

lose  $B  if  event  E  does  not  occur 

Bet  1  b  -  win  $A  with  probability  of  q 

lose  SB  with  probability  of  \~q. 

The  expected  values  of  bets  la  and  1  b  to 
the  expert  are  respectively  Ap  +  Bp-  B  and/h/ 
+  Bq  -  B,  where  Pis  the  probability  of  event  E 
occurring.  The  following  inferences  may  be 
drawn  from  the  experts  decision:  if  bet  la  is 
chosen,/!/?  +  Bp-B  >_Aq  +  Bq  •  /?,  so/?  _>.</; 
likewise  if  1  b  is  selected  p  _<  <7. 

By  repeating  the  procedure,  varying  the 
value  of  q,  the  probability  of  event  E  can  be 
ascertained.  It  is  the  point  at  which  the  expert  is 
indifferent  between  bets  la  and  lb,  where  p  = 
q.  The  degree  of  precision  is  dependent  on  the 
number  of  bets  and  the  incremental  changes  of 
the  value  of  q. 
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PROBABILITY 


A  way  of  avoiding  the  problem  of  a 
large  number  of  bets  to  obtain p  would  be  to  as¬ 
sess  the  probabilities  through  the  use  of  direct 
interrogation,  and  then  to  use  the  betting  situ¬ 
ation  as  a  check  on  the  assumed  probabilities. 

To  complete  a  PDF,  the  analyst  repeats 
this  procedure  over  a  relevant  range  of  interval 
values.  The  analyst  then  plots  the  points  at  the 
center  of  the  range  for  each  event  and 
smoothes  in  a  curve,  so  that  the  area  under  it 
equals  one,  as  in  Figure  F-2.  The  analyst  must 
ensure  that  all  of  the  relevant  axioms  of  prob¬ 
ability  are  maintained. 


Figure  F-2  Fitting  a  Curve  to 
Expert  Judgment 

Many  people,  when  questioned  one 
way,  are  likely  to  make  probability  statements 
that  are  inconsistent  with  what  they  will  say 
when  questioned  in  another  equivalent  way,  es¬ 
pecially  when  they  are  asked  for  direct  assign¬ 
ment  of  probabilities.  As  the  number  of  events 
increases,  so  does  the  difficulty  of  assigning  di¬ 
rect  probabilities.  Therefore,  when  this  is  a 


problem,  the  betting  method  is  most  appropri¬ 
ate. 

To  apply  the  betting  technique,  we  will 
select  one  interval  for  the  relevant  range  to 
demonstrate  how  this  method  can  be  used  to 
obtain  probability  estimates  and,  hence,  PDFs. 
The  bet  is  established  as  follows: 

Bet  la  -  win  $10,000  if  cost  is  between 

$15,100  and  $20,000 
lose  $5,000  if  cost  is  not 
between  $15,100  and  $20,000 

Bet  1  b  -  win  $10,000  with  probability 

of  q 

lose  $5,000  with  probability 
of  l-<? 

The  value  of  q  is  established  initially, 
and  the  expert  is  asked  which  of  the  two  bets 
he  would  take. 

The  value  of  q  is  then  varied  systemati¬ 
cally,  either  increased  or  decreased.  The  point 
at  which  the  expert  is  indifferent  between  the 
two  bets  (with  the  associated  q  value)  provides 
the  probability  of  the  cost  being  between 
$15,100  and  $20,000.  This  process  is  repeated 
for  each  interval,  and  the  results  used  create 
the  PDF  associated  with  the  cost  of  that  par¬ 
ticular  program  event. 

4)  Modified  Churchman/ 

Ackoff  Technique 

Another  method,  which  can  be  used  to 
ascertain  PDFs  for  cost,  schedule,  or  perform¬ 
ance  parameters,  is  the  “Modified  Church- 
man-Ackoff  method."  This  technique  builds 
upon  procedures  which  were  presented  by 
Churchman  and  Ackoff  in  1954.  This  tech¬ 
nique  was  developed  as  a  means  to  order 
events  in  terms  of  likelihood.  The  modification 
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to  the  technique  was  performed  so  that  once 
the  order  of  event  likelihoods  had  been  accom¬ 
plished,  relative  probabilities  could  be  as¬ 
signed  to  the  events  and  finally  probability 
density  functions  developed.  So  as  to  be  rele¬ 
vant  for  our  purposes,  events  are  defined  as 
range  values  for  cost,  schedule,  or  perform¬ 
ance  (activity  durations)  relating  to  the  out¬ 
come  of  a  specific  activity  in  a  program. 

The  modified  Churchman- Ackoff  tech¬ 
nique  is  most  appropriate  when  there  is  one  ex¬ 
pert,  and  that  expert  has  a  thorough 
understanding  of  the  relative  ranking  of  cost/ 
schedule  ranges  and  a  limited  understanding 
of  probability  concepts.  The  remainder  of  this 
section  was  extracted  and  modified  from  the 
Compendium  on  Risk  Analysis  Techniques 
(1972,  see  references).  Note  that  while  the 
mathematical  calculations  appear  to  make  this 
a  very  precise  technique,  it  is  still  an  approxi¬ 
mation  of  an  expert’s  judgment  and  should  not 
be  interpreted  to  be  more  exact  than  other 
similar  techniques. 

The  first  step  in  applying  the  modified 
Churchman-Ackoff  technique  is  to  define  the 
relevant  range  of  values.  That  is,  the  end 
points,  along  a  range  of  values  with  zero  prob¬ 
ability  of  occurrence  must  be  specified.  These 
values  need  only  be  any  low  and  high  values 
which  the  expert  specifies  as  having  zero  prob¬ 
ability  of  occurrence.  Next,  ranges  of  individual 
values  within  the  relevant  range  must  be  deter¬ 
mined.  These  ranges  of  values  which  will  form 
the  set  of  comparative  values  for  this  technique 
are  specified  by  the  following  approach: 


(1)  Start  with  the  low  value  in 
the  relevant  range 

(2)  Progress  upward  on  the 
scale  of  values  until  the  ex¬ 
pert  is  able  to  state  a  simple 
preference  regarding  the 
relative  probabilities  of  oc¬ 
currence  of  the  two  charac¬ 
teristic  values.  If  he  is  able 
to  say  that  he  believes  one 
value  has  either  a  greater 
chance  or  a  lesser  chance  of 
occurring  that  the  other  of 
the  two  values,  then  it  is  in¬ 
ferred  that  the  expert  is  able 
to  discriminate  between  the 
two  values. 

(3)  Using  the  higher  of  the  two 
previously  specified  scale 
values  as  a  new  basis,  repeat 
step  (2)  to  determine  the 
next  value  on  the  scale. 

(4)  Repeat  steps  (2)  and  (3)  un¬ 
til  the  high  end  point  value 
of  the  range  of  parameters 
values  is  approached. 


Employing  this  procedure  for  the  dura¬ 
tion  required  to  successfully  test  a  piece  of 
equipment,  may  yield  the  results  shown  in  Ta¬ 
ble  F-l. 


Table  F-l  Characteristic  Values  for 
Equipment  Test  Durations 


Oi 

0-3 

days 

02 

= 

4-7 

days 

03 

— 

8-11 

days 

04 

= 

12-15 

days 

05 

= 

16  -  19 

days 

0e 

= 

20  -  23 

days 

07 

24-  21 

days 

The  descending  order  of  probability  or 
occurrence  can  be  determined  by  applying  the 
following  paired  comparison  method. 
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Table  F-3 


Transformation 


Ask  the  expert  to  compare,  one  at  a 
time,  the  first  interval  value  (Oi)  of  the  set  to 
each  of  the  other  values  (O2, 03,  etc.),  stating  a 
preference  for  that  value  in  each  group  of  two 
values  that  he  believes  has  the  greater  chance 
of  occurring  (denoting  a  greater  probability  of 
occurrence  by  > ,  and  equal  chance  by  = ,  and  a 
lesser  chance  by  <).  The  following  hypotheti¬ 
cal  preference  relationships  could  result  for  a 
set  of  seven  values  (Oi  <  O2,  Oi  <  O3,  Oi  <  O4, 
Oi  <  O5,  Oi  <  06,  Oi  <  O7). 

Next,  ask  the  expert  to  compare,  one  at 
a  time,  the  second  interval  values  (O2)  of  the  set 
to  each  of  the  other  interval  values  succeeding 
it  in  the  set  (i.e.,  O3,  O4,  etc.).  The  following 
preference  relationships  might  result  (O2  <  O3, 
O2  <  O4, 02  <  O5,  O2  <  06,  O2  <  O7).  Con¬ 
tinue  this  process  until  all  values  (0j )  have 
been  compared. 

Now  total  the  number  of  times  (0j ) 
value  was  preferred  over  other  values.  The  re¬ 
sults  for  this  procedure  are  listed  in  Table  F-2. 

Table  F-2  Summary  of  Preference 
Relationships 


04 

s= 

6  times 

03 

= 

5  times 

05 

ss 

4  times 

02 

= 

3  times 

06 

= 

2  times 

0, 

= 

0  times 

07 

= 

0  times 

List  the  values  in  descending  order  of 
simple  ordinal  probability  preference  and 
change  the  symbols  for  each  value  from  0j  to 
Xj  as  shown  in  Table  F-3. 


CHARACTERISTIC 

VALUE 

(DAYS) 

PREFERENCE 

RANK 

NEW 

SYMBOL 

0-3 

0< 

1 

X, 

4-7 

o3 

2 

X2 

8-11 

o5 

3 

X3 

12-15 

o2 

4 

X4 

16-19 

0e 

5 

X5 

20-23 

0, 

6 

X, 

24-27 

Or 

7 

X7 

Arbitarily  assign  a  rating  of  100  points 
to  the  characteristic  value  with  the  highest 
subjective  probability  (e.g.,  Xi).  Then,  as  in  the 
first  step,  question  the  expert  regarding  the 
relative  chance  of  occurrence  of  each  of  the 
other  values  on  the  ordinal  scale  in  Table  F-3 
with  respect  to  the  value  at  the  top  of  the  scale. 
Assigning  Xi  a  rating  of  100  points,  the  expert 
is  first  interrogated  as  to  his  feeling  of  the  rela¬ 
tive  chance  of  occurrence  of  the  second  highest 
scale  value  (e.g.,  X2),  with  respect  to  Xi.  Does 
it  have  25  percent  chance?  60  percent?  70  per¬ 
cent?  80  percent?  As  much  chance  of  realiza¬ 
tion  as  Xi?  The  relative  probability  rating, 
based  on  100  points,  (i.e.,  100  percent  as  much 
chance)  will  then  be  posted  for  X2. 

Next,  question  the  expert  about  the 
relative  chance  of  occurrence  of  the  next  high¬ 
est  scale  (e.g.,  X3)  first  with  respect  to  the  most 
preferred  value  (Xi),  and  then  with  respect  to 
the  second  most  preferred  scale  value  (X2). 
The  resulting  numerical  ratings  should  concur. 
For  example,  if  the  expert  decides  that  X2  has 
8/10  as  much  chance  of  occurring  as  does  Xi, 
and  that  X3  has  1/2  as  much  chance  as  Xi,  and 
5/8  as  much  chance  as  X2.  the  ratings  become 
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Xi  =  100  points,  X2  =  80  points,  and  X3  =  50 
points. 


This  process  continues  for  each  succes¬ 
sively  lower  interval  value  on  the  ordinal  scale 
as  shown  in  Table  F-3.  Determine  the  relative 
number  of  points  to  be  accorded  each  value 
with  respect  to  the  top  scale  and  with  respect  to 
all  other  values  on  down  the  scale  which  are 
above  the  characteristic  value  in  question. 

In  the  event  of  minor  disparities  be¬ 
tween  relative  probability  ratings  for  a  given 
value,  the  average  of  all  such  ratings  for  that 
characteristic  value  might  be  computed  For 
example,  X4  might  be  determined  to  be  3/10  as 
probable  as  Xi,  1/4  as  probable  as  X2,  and  1/2 
as  probable  as  X3.  The  three  absolute  ratings 
for  X4  are  thus  inferred  to  be  30,  20,  and  25 
points,  respectively.  The  average  of  these  rat¬ 
ings  is  25.  However,  before  averaging  such  fig¬ 
ures,  it  might  be  beneficial  to  have  the  expert 
revaluate  his  relative  ratings  for  X4  with  re¬ 
spect  to  Xi,  X2,  and  X5. 

As  a  result  of  the  above  process,  the 
relative  probability  values  shown  in  Table  F-4 
might  be  attained. 


actual  probability  density  values  by  letting 
P(Xi)  equal  the  actual  subjective  probability 
or  occurrence  of  the  highest  value.  Then,  P(X2) 
is  then  defined  as: 


R(X2) 

R(Xj) 


[P(XX )] 


Similarly  P(Xj )  is  defined  as: 


R(Xj) 

R(Xj) 


[P(Xj  )1 


for  i  =  2,  3, ....  7. 

Assuming  that  the  independent  charac¬ 
teristic  values  evaluated  represent  all  possible 
values  attainable  by  the  component  character¬ 
istic,  the  respective  probabilities  must  sum  to 
1.0  (i.e.,  P(Xi)  +  P(X2)  +  P(X3)  +  P(X4)  + 
P(X5)  +  P(X6)  +  P(X7)  -  1.0).  Substituting 

the  expressions  for  P(X  j ),  i  -  2 . 7,  it  follows 

that: 


Table  F-4  Relative  Probability  Ratings 


RX, 

cs 

100  Probability  points 

rx2 

- 

80  Probability  points 

rx3 

» 

50  Probability  points 

RX4 

= 

25  Probability  points 

RXS 

= 

10  Probability  point:. 

RXe 

0  Probability  points 

RX7 

=s 

0  Probability  points 

Finally,  the  scale  of  relative  probability 
values  can  be  converted  directly  into  a  scale  of 


Solving  this  equation  for  P(Xi),  the  remaining 
P(X  j ),  i  =  2, ....  7  can  be  determined  using  the 
relationship : 

R(Xj) 

As  an  illustration,  consider  the  relative 
probability  ratings  in  Table  F-4.  Using  the 
values,  the  preceding  equation  is  given  by: 
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rcxp  ♦  1  P(x,)  +  $  PC*,)  - 
ii  e^i)  +  M  W  ’  L 

Solving  this  equation,  P(Xi)  =  0.377. 

This  value  can  be  used  to  determine  the 
remaining  probabilities  as  follows: 

P (X2)  -  H|  P(Xi)  »  0.80(0.377)  =  0.301 
P(X3)«||1  P(Xi)  «  0.50(0.377)  =  0.189 
P(X4)  «  Hi  P(X1)  -  0.25  (0.377)  *  0.095 

K/v  j 

P(X5)»Hi  P(Xi)  -  0.10(0.377)  =  0.038 
J  RX  2 

P(X6)-Hi  P(XD  ■  0(0.377)  =  0.000 

P(X7)  *  Hi  P(Xi)  «  0  (0.377)  -  0.000 

The  resulting  probability  density  ap> 
pears  in  Table  F-5. 


Table  F-5  Probability  Density 


COMPONENT 

CHARACTERISTIC 

VALUE 

PROBABILITY 

X, 

0.377 

Xt 

0.301 

X, 

0.189 

X. 

0.095 

X, 

0.038 

X. 

0.000 

X, 

0.000 

TOTAL  1.000 

5)  Delphi  Approach 

In  many  cases,  expert  judgment  does 
not  reside  solely  with  one  individual,  but  is 
spread  among  multiple  experts.  Committee  ap¬ 
proaches  to  obtaining  a  group  assessment  have 
been  found  to  contain  problems  relating  to  in¬ 
terpersonal  pressures  to  a  degree  that  caused 
researchers  at  the  RAND  Corporation  to  de¬ 
vise  a  method  known  as  the  Delphi  to  avoid  the 
pressures. 

The  Delphi  technique  has  become  well 
known  in  management  circles,  but  is  subject  to 
misconception.  Too  often  the  term  is  used  to 
identity  a  committee  or  multiple  interview 
process,  and  these  do  not  share  the  advantages 
of  the  Delphi  technique. 

The  Delphi  technique  has  been  ex¬ 
tended  in  recent  years  to  cover  a  wide  variety 
of  types  of  group  interaction.  The  technique 
can  be  used  for  group  estimation,  that  is,  the 
use  of  a  group  of  knowledgeable  individuals  to 
arrive  at  an  estimate  of  an  uncertain  quantity. 
The  quantity  can  be  a  cost,  a  time  period  asso¬ 
ciated  with  an  event,  or  a  performance  level. 

The  Delphi  technique  is  most  appropri¬ 
ate  when: 

•  The  problem  does  not  lend 
itself  to  precise  analytical 
techniques  but  can  benefit 
from  subjective  judgments 
on  a  collective  basis. 

•  The  individuals  needed  to 
contribute  to  the  examina¬ 
tion  of  a  broad  or  complex 
problem  have  no  history  of 
adequate  communication 
and  may  represent  diverse 
backgrounds  with  respect 
to  experience  or  expertise. 


•  More  individuals  are 
needed  than  can'  effectively 
interact  in  a  face-to-face 
exchange. 

•  Time  and  cost  make  fre¬ 
quent  group  meetings  un¬ 
feasible. 

•  The  efficiency  of  face-to- 
face  meetings  can  be  in¬ 
creased  by  a  supplemental 
group  communication 
process. 

•  Disagreements  among  indi¬ 
viduals  are  so  severe  or  po¬ 
litically  unpalatable  that 
the  communication  process 
must  be  refereed  and/or 
anonymity  assured. 

•  The  heterogeneity  of  the 
participants  must  be  pre¬ 
served  to  assure  validity  of 
the  results,  i.e.,  avoidance 
of  domination  by  quantity 
or  by  strength  of  personality 
(“bandwagon  effect”). 


The  Delphi  technique  differs  from 
other  methods  of  obtaining  a  group  opinion, 
because  it  physically  separates  the  group’s 
members  from  one  another  in  order  to  reduce 
irrelevant  interpersonal  influences.  Properly 
carried  out,  the  technique  is  facilitated  by  an 
analyst  obtaining  each  panel  member’s  opinion 
and  each  member’s  reason  for  the  opinion.  The 
analyst  then  reduces  the  opinions  and  reasons 
to  standard  statements  in  order  to  preserve 
anonymity.  The  analyst  then  shows  the  panel 
member  the  aggregated  opinions  of  the  other 
panel  members  in  statistical  terms.  The  analyst 
provides  each  panel  member  with  the  reasons 
justifying  the  opinions  that  differ  with  the 
member,  and  requests  revaluation  and  further 
substantiation.  This  iterative  feeding  back  con¬ 


tinues  until  no  further  substantial  change  re¬ 
sults.  At  this  point,  the  moderator  takes  the 
final  individual  opinions  and  computes  a  set  of 
median  values  to  represent  the  group  opinion. 
The  median  value,  rather  than  the  average,  is 
used  as  a  central  estimate  to  prevent  the  esti¬ 
mate  from  being  overly  influenced  by  extreme 
individual  values. 

One  technique  which  hold  much  prom¬ 
ise  for  the  future  as  a  means  of  capturi  ng  expert 
judgment  is  “expert  support  systems”.  Ideally, 
the  expert  support  system  would  lead  the  ex¬ 
perts)  through  a  series  of  parameter  specific 
questions  (cost  and  schedule,  possibly  per¬ 
formance)  and  generate  PDFs  based  on  the  re¬ 
sponses. 


III.  RESOURCE  REQUIREMENTS 

The  effort  required  to  conduct  expert 
interviews  and  generate  appropriate  PDFs  is 
man-hour  intensive.  Much  time  is  spent  by  the 
analyst  with  the  expert(s)  acquiring  and  quanti¬ 
fying  his  expertise.  The  amount  of  time  re¬ 
quired  to  accomplish  this  task  is  predicated  on 
the  number  of  PDFs  needed  (based  on  the 
numbei  of  activities  required  as  model  input 
and  whether  cost,  schedule,  and  technical  dis¬ 
tributions  are  requited).  The  methods  de¬ 
scribed  are  basically  manual  with  computer 
resources  not  a  necessity.  However,  as  the  tech¬ 
niques  become  more  complex  and  expert  sup¬ 
port  systems  to  accomplish  the  tasks  are 
developed,  computer  resources  required  will 
escalate  dramatically. 
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IV.  RELIABILITY 

The  reliability  of  the  PDFs  obtained 
through  these  techniques  is  affected  by  a  num¬ 
ber  of  factors.  Foremost  is  the  degree  to  which 
the  so  called  “expert”  is  in  fact  an  expert.  The 
better  understanding  the  expert  has  of  the  pa¬ 
rameter  being  modeled,  the  more  reliable  the 
resulting  PDFs  will  be.  The  burden  also  falls 
on  the  analyst  to  select  the  technique  most  ap¬ 
propriate  for  obtaining  PDFs.  For  example,  if 
expertise  resides  with  more  than  one  expert,  a 
Delphi  technique  would  result  in  much  more 
reliable  PDFs  than  would  a  direct  method  of 
asking  only  one  expert.  Likewise,  if  the  expert 
has  very  little  understanding  of  probability 
concepts,  it  would  be  inappropriate  to  ask  him 
to  select  a  PDF  from  a  visual  list  of  options. 
Under  these  circumstances,  the  modified 
Churchman- Ackoff  method  or  a  betting  tech¬ 
nique  would  most  likely  result  in  more  reliable 
PDFs.  In  summary,  much  of  the  reliability  of 
the  PDFs  is  predicated  on  the  techniques  se¬ 
lected  by  the  analyst  for  constructing  them. 
Therefore,  it  is  important  that  the  analyst  know 
when  each  technique  is  most  appropriate, 
given  the  unique  circumstances  of  that  specific 
program  office. 

V.  Sources  of  Additional  Information 

Singleton,  W.T.  &  Houden,  J.,  “Risk  &  Deci¬ 
sion”,  1987,  John  Wiley  &  Sons  Ltd. 

DeGroot,  M.H.,  “Optimal  Statistical  Deci¬ 
sions”,  1970,  New  York,  McGraw  Hill. 


Winkler,  R.L.,  (1967)  “The  Quantification  of 
Judgment:  Some  Methodological  Sugges¬ 
tions,”  Journal  of  the  American  Statistical  As¬ 
sociation  62, 1105-1120. 


Winkler,  R.L.,  (1971)  “Probabilistic  Predic¬ 
tion:  Some  Experimental  Results,”  Journal  of 
the  American  Statistical  Association  66, 
675-685. 


The  Delphi  Method  Techniques  and  Applica¬ 
tions,  Linstone,  H.A.,  Turoff,  M.,  1975  Ad- 
dison-Wesley  Publishing  Company,  Reading, 
MA. 


Dalkey,  Norman  C.,  “The  Delphi  Method:  An 
Experimental  Study  of  Group  Opinion”,  The 
RAND  Corp.,  Santa  Monica,  CA,  1968. 


Brown,  R.V.,  Kahr,  A.S.S.,  and  Peterson,  C., 
“Decision  Analysis  for  the  Manager”,  Halt, 
Rinehart  &  Winston,  New  York,  NY,  1974. 


Atzinger,  E.M.  et  al.  Compendium  on  Risk 
Analysis  Techniques,  DARCOM  Material  Sys¬ 
tems  Analysis  Activity,  Aberdeen  Proving 
Ground,  MD  1972,  (AD  746245,  LD  28463). 
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APPENDIX  G 

SPECIAL  NOTES  ON  SOFTWARE  RISK 


G-l 


While  the  techniques  and  processes  dis¬ 
cussed  in  the  text  of  the  guide  do  apply  to  soft¬ 
ware,  they  do  not  address  some  of  the 
peculiarities  that  are  a  part  of  software  devel¬ 
opment.  Software  has  a  tendency  to  change 
dramatically  during  the  development  cycle 
when  compared  to  hardware.  This  brief  appen¬ 
dix  is  intended  to  generate  some  thought  and 
suggest  some  useful  actions  in  managing  soft¬ 
ware  development  efforts.  Additional  informa¬ 
tion  can  be  obtained  from  Chapter  20  of  the 
DSMC  Systems  Engineering  Management 
Guide. 

One  of  the  most  effective  risk  manage¬ 
ment  (handling)  techniques  for  software  is  the 
establishment  of  a  formal  Software  Quality  As¬ 
surance  program  early  in  the  development  cy¬ 
cle.  The  program  should  establish  a  “team”  of 
experts  whose  charter  is  to  explicitly  look  at  is¬ 
sues  which  will  ensure  a  reliable  product  in  a 
reasonable  time  and  at  a  reasonable  cost. 
Some  of  the  issues  that  the  team  must  evaluate 
include  the  following: 

•  Is  independent  verification 
and  validation  warranted 

•  Is  the  development  envi¬ 
ronment  adequate 

-  tool  sets 

-  compiler 

•  Is  the  higher  order  lan¬ 
guage  selection  appropri¬ 
ate 

•  Are  the  requirements 
clearly  stated 

•  Will  rapid  prototyping  be 
used 

•  Has  the  software  approach 
been  baselined 


•  Has  the  testing  philosophy 
been  established 

•  Has  the  development  phi¬ 
losophy  been  established? 

Addressing  these  issues  early  in  the  de¬ 
velopment  cycle  will  help  avoid  surprises 
downstream.  There  are  three  other  documents 
that  may  provide  useful  in  software  risk  man¬ 
agement: 

AFSCP  800-XX  (Draft),  Air  Force  System 
Command  Software  Risk  Management.  Jun 
87. 

ASD  Pamphlet  800-5  Acquisition  Manage¬ 
ment,  Software  Development  Capability/Ca¬ 
pacity  Review,  10  Sep  87. 

Software  Reporting  Metrics,  Electronic  Sys¬ 
tems  Division,  AFSC,  Hanscom  AFB,  MA. 
Nov  85. 

These  documents  contain  more  specific 
actions  for  dealing  with  software  development 
problems.  The  basic  process  for  risk  manage¬ 
ment  still  applies  to  software  -  plan,  assess, 
analyze,  and  handle.  Tables  G- 1  to  G-5  are  ex¬ 
tracts  from  the  draft  AFSC  pamphlet  that  may 
prove  useful  in  quantifying  software  risk. 

The  “Software  Reporting  Metrics" 
document  has  proven  extremely  useful  and 
both  the  Army  and  Air  Force  have  issued  for¬ 
mal  guidance  regarding  the  use  of  this  tech¬ 
nique  in  AMC  Pamphlet  70-13  and  AFSCP 
800-43. 
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Table  G-l  Quantification  of  Probability  and 
Impact  of  Technical  Failure 


MAGNITUDE 

TECHNICAL  DRIVERS 

LOW 

(0.0  -  0.3) 

MEDIUM 

HIGH 
(0.6-  1.0) 

REQUIREMENTS 

COMPLEXITY 

Simple  or  easily 
allocatable 

Moderate,  can  be  al¬ 
located 

Significant  or 
difficult  to  allocate 

SIZE 

Small  or  easily  broken 
down  into  work  units 

Medium,  or  can  be  broken 
down  Into  work  units 

Large  or  cannot  be  broken  down 
into  work  loads 

STABILITY 

Little  or  no  change 
to  establisned  baseline 

Some  change  In 
baseline  expected 

Rapidly  changing  or 
no  baseline 

PDSS 

Agreed  to  support 
concept 

Roles  and  missions 

Issues  unresolved 

No  support  concept  or 
major  unresolved  issues 

R&M 

Allocatable  to  hardware 
and  software  components 

Requirements  can 
be  defined 

Can  only  be  addressed 
at  the  total  system  level 

CONSTRAINTS 

COMPUTER  RESOURCES 

Mature,  growth  capacity 
within  design,  flexible 

Available,  some 
growth  capacity 

New  development  no 
growth  capacity,  inflexible 

PERSONNEL 

Available,  in  place, 
experienced,  stable 

Available,  but  not  in 
place,  some  experience 

High  turnover,  little  or  no 
experience,  not  available 

STANDARDS 

Appropriately  tailored 
for  application 

Some  tailoring,  all  not 
reviewed  for  applicability 

No  tailoring,  none  applied 
to  the  contract 

GFE/GFP 

Meets  requirements, 
available 

May  meet  requirements, 
uncertain  availability 

Not  compatible  with  system 
requirements,  unavailable 

ENVIRONMENT 

Little  or  no  impact  on 
design 

Some  impact 
on  design 

Major  impact 
on  design 

TECHNOLOGY 

LANGUAGE 

Mature,  approved 

HOL  used 

Approved  or 

N on-approved  HOL 

Significant  use  of 
assembly  language 

HARDWARE 

Mature,  available 

Some  development 
or  available 

Total  new  de¬ 
velopment 

TOOLS 

Documented,  validated, 
in  place 

Available,  validated 
some  development 

Unvalidated,  proprietary, 
major  development 

DATA  RIGHTS 

Fully  compatible  with 
support  and  follow-on 

Minor  incompatibilities 
with  support  and  follow-on 

Incompatible  with  support 
and  follow-on 

EXPERIENCE 

Greater  than  3  to  5  years 

Less  than  3  to  5  years 

Little  or  none 

DEVELOPMENTAL 

APPROACH 

PROTOTYPES  &  REUSE 

Used,  documented 
sufficiently  for  use 

Some  use  and 
documentation 

No  use  and/or 
no  documentation 

DOCUMENTATION 

Correct  and 
available 

Some  deficiencies, 
available 

Nonexistent 

ENVIRONMENT 

In  place,  validated, 
experience  with  use 

Minor  modifications, 
tools  available 

Major  development 
effort 

MANAGEMENT  APPROACH 

Existing  product  a  no 
process  controls 

Product  &  process  controls 
need  enhancement 

Weak  or 
nonexistent 

INTEGRATION 

Internal  and  external 
controls  >n  olace 

Internal  or  external 
controls  not  in  Dlace 

Weak  or 
nonexistent 

IMPACT 

Minimal  to  small  reduction 
in  technical  performance 

Some  reduction  in 
technical  performance 

Significant  degredation 
to  nonachievement  of 
technical  performance 

Table  G-2  Quantification  of  Probability  and 
Impact  of  Technical  Failure 


“  MAGNITUDE 

OPERATIONAL  DRIVERS 

LOW 

(0.0  -  0.3) 

MEDIUM 
(0.4 -0.5) 

HIGH 
(0.6 -1.0) 

USER  PERSPECTIVE 

REQUIREMENTS 

Compatible  with  the 
user  environment 

Some  Incompatibilities 

Major  incompatibilities 
with  “ops'  concepts 

STABILITY 

Little  or  no  change 

Some  controlled  change 

Uncontrolled  change 

TEST  ENVIRONMENT 

Representative  ot  the 
user  environment 

Some  aspects  are 
not  representative 

Major  disconnects  with 
user  environment 

OTAE  RESULTS 

Test  errors /failures 
are  correctable 

Some  error  s/failures  are 
not  correctable  before  IOC 

Major  corrections 
necessary 

QUANTIFICATION 

Primarily  objective 

Some  subjectivity 

Primarily  subjective 

TECHNICAL 

PERFORMANCE 

USABILITY 

User  friendly 

Mildly  unfriendly 

User  unfriendly 

RELIABILITY 

Predictable  performance 

Some  aspects 
unpredictable 

Unpredictable 

FLEXIBILITY 

Adaptable  with  threat 

Some  aspects  ara 
not  adaptable 

Critical  functions 
not  adaptable 

SUPPORTABILITY 

Timely  incorporation 

Response  times 
inconsistent  with  need 

Unresponsive 

INTEGRITY 

Responsive  to  update 

Hidden  linkages, 
controlled  access 

Insecure 

PERFORMANCE 

ENVELOPE 

ADEQUACY 

Full  compatibility 

Some  limitations 

Inadequate 

EXPANDABILITY 

Easily  expanded 

Can  be  expanded 

No  expansion 

ENHANCEMENTS 

Timely  incorporation 

Some  lag 

Major  delays 

THREAT 

Responsive  to  change 

Cannot  respond 
to  some  changes 

Unresponsive 

IMPACT 

Full  mission 
capability 

Some  limitations 
on  mission 
performance 

Severe 

performance 

limitations 
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Table  G-3  Quantification  of  Probability  and 
Impact  of  Support  Failure 


MAGNITUDE 

SUPPORT  DRIVERS 

LOW 

(0.0  -  0.3) 

MEDIUM 
(0.4  -  0.5) 

HIGH 
(0.6-  1.0) 

DESIGN 

COMPLEXITY 

Structurally 

maintainable 

Certain  aspects 
difficult 

Extremely  difficult 
to  maintain 

DOCUMENTATION 

Adequate 

Some  deficiencies 

Inadequate 

COMPLETENESS 

Little  additional  for 

PDSS  incorpora  ion 

Some  PDSS 
incorporation 

Extensive  PDSS 
incorporation 

CONFIGURATION  MANAGEMENT 

Sifficlent,  In 
place 

Some  shortfalls 

Insufficient 

STABILITY 

Little  or  no  change 

Moderate,  controlled 
change 

Rapid  or  uncontrolled 
change 

RESPONSIBILITIES 

MANAGEMENT 

Defined,  assigned 
responsibilities 

Sc.ne  roles  and 
mission  issues 

Undefined  or 
unassigned 

CONFIGURATION  MANAGEMENT 

Single  point  control 

Defined  jontrol  points 

Multiple  control 
points 

TECHNICAL  MANAGEMENT 

Consistent  with 
operational  needs 

Some  inconsistencies 

Major  inconsistencies 

CHANGE  IMPLEMENTATION 

Responsive  to 
user  needs 

Acceptable  delays 

..  .....  ....  .  .  ..  _  ...  . 

Nonresponsive  to 
user  needs 

TOOLS  &  MANAGEMENT 

FACILITIES 

In  place,  little 
change 

In  place,  some 
modification 

Nonexistent  or 
extensive  change 

SOFTWARE  TOOLS 

Delivered,  certified, 
sufficient 

Some  resolvable 
concerns 

Not  delivered,  certified, 
or  sufficient 

COMPUTER  HARDWARE 

Compatible  with 
*ops‘  system 

Minor 

incompatibilities 

Major  incompatibilities 

PRODUCTION 

Sufficient  for 
fielded  units 

Some  capacity 
questions 

Insufficient 

DISTRIBUTION 

Controlled,  responsive 

Minor  response 
concerns 

Uncontrolled  or 
nonresponsive 

SUPPORTABILITY 

CHANGES 

Within  projections 

Slight  deviations 

Major  deviations 

OPERATIONAL  INTERFACES 

Defined.controlled 

Some  ‘hidden’ 
linkages 

Extensive  linkages 

PERSONNEL 

In  place,  sufficient, 
expenence 

Minor  discipline 
mix  concerns 

Significant  concerns 

RELEASE  CYCLE 

Responsive  to 
user  requireme>  ts 

Minor 

incompatibilities 

Nonresponsive  to 
user  needs 

PROCEDURES 

In  place,  adequate 

Some  concerns 

Nonexistent  or 
inadequate 

IMPACT 

Responsive  software 
support 

Minor  delays  in 
software  modifications 

Nonresponsive  or 

unsupportable 

software 

Table  G-4  Quantification  of  Probability  and 
Impact  of  Support  Failure 


~ “  MAGNITUDE 

COST  DRIVERS 

LOW 

(0.0  -  0.3) 

MEDIUM 
(0.4  -  0.5) 

HIGH 
(0.6-  1.0) 

REQUIREMENTS 

SIZE 

Small,  non-complex,  or 
easily  decomposed 

Medium,  moderate 
complexity,  decomposable 

Large,  highly  complex, 
or  not  decomposable 

RESOURCE  CONSTRAINTS 

Little  or  no  hardware 

Imposed  constraints 

Some  hardware 

Imposed  constraints 

Significant  hardware 
imposed  constraints 

APPLICATION 

Non  real-time,  little 
system  interdependency 

Embedded,  some 
system  Interdependency 

Real-time,  embedded, 
strong  interdependency 

TECHNOLOGY 

Mature,  existent,  In- 
house  experience 

Existent,  some  in- 
house  experience 

New  or  new  application, 
little  experience 

REQUIREMENTS  STABILITY 

Little  cr  no  change 
to  established  baseline 

Some  change  In 
baseline  expected 

Rapidly  changing  or 
no  baseline 

PERSONNEL 

AVAILABILITY 

In  place,  little 
turnover  expected 

Available,  some 
turnover  expected 

High  turnover,  not 
available 

MIX 

Good  mix  of  software 
disciplines 

Some  disciplines 

Inappropriately  represented 

Some  disciplines 
not  represented 

EXPERIENCE 

High  experience  ratio 

Average  experience 
ratio 

Low  experience  ratio 

MANAGEMENT 

ENGINEERING 

Strong  management 
approach 

Good  personnel 
management  approach 

Weak  personnel 
management  approach 

REUSABLE  SOFTWARE 

AVAILABILITY 

Compatible  with 
need  dates 

Delivery  dates  in 
question 

Incompatible  with 
need  dates 

MODIFICATIONS 

Little  or  no 
change 

Some  change 

Extensive  changes 

LANGUAGE 

Compatible  with  system 

4  PDSS  requirements 

Partial  compatibility 
with  requirements 

Incompatible  with  system 
or  PDSS  requirements 

RIGHTS 

Compatible  with  PDSS 
&  competition  requirements 

Partial  compability  with 

POSS,  some  competition 

Incompatible  with  PDSS 
concept,  noncompetitive 

CERTIFICATION 

Verified  performance, 
application  compatible 

Some  application  compatible 

PDSS,  some  competition 

Unverified,  little  test 
data  available 

TOOLS  AND 
ENVIRONMENT 

FACILITIES 

Little  or  no 
modifications 

Seme  modificastions. 
existent 

Major  modifications, 
nonexistent 

AVAILABILITY 

In  place,  meets 
need  dates 

Some  compatibility 
with  need  dates 

Nonexistent,  does  not 
meet  need  dates 

RIGHTS 

Compatible  with  PDSS 

4  development  plans 

Partial  compatality  with 

PDSS  4  development  plans 

Incompatible  with  PDSS 

4  development  plans 

CONFIGURATION 

MANAGEMENT 

Fully  controlled 

Some  controls 

No  controls 

IMPACT 

Sufficient  financial 
resources 

Some  shortage  of 
financial  resources, 
possible  overrun 

Significant  financial 
shortages,  budget 
overrun  likely 
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Table  G-5  Quantification  of  Probability  and 
Impact  of  Schedule  Failure 


MAGNITUDE  j 

SCHEDULE  DRIVERS 

LOW 

(0.0  -  0.3) 

MEDIUM 
(0.4  -  0.5) 

HIGH 
(0.6-  1.0) 

RESOURCES 

PERSONNEL 

Good  discipline  mix 
in  place 

Some  disciplines 
not  available 

Questionable  mix 
and/or  availability 

FACILITIES 

Existent,  little  or  no 
modification 

Existent  some 
modification 

Nonexistent, 
extensive  changes 

FINANCIAL 

Sufficient  budget 
allocated 

Some  questionable 
allocations 

Budget  allocation 
in  doubt 

NEED  DATES 

THREAT 

Verified  Projections 

Some  unstable 
aspects 

Rapidly  changing 

ECONOMIC 

Stable  commitments 

Some  uncertain 
commitments 

Unstable,  fluctuating 
commitments 

POLITICAL 

Little  projected 
sensitivity 

Some  limited 
sensitivity 

Extreme  sensitivity 

GFE/GFP 

Available,  certified 

Certification  or 
delivery  questions 

No  application 
evidence 

TOOLS 

In  place,  available 

Some  deliveries 
in  question 

Little  or  none 

TECHNOLOGY 

AVAILABILITY 

In  place 

Baselined, 
some  unknowns 

Unknown,  no 
baseline 

MATURITY 

Application  verified 

Controllable  change 
projected 

.  Rapid  or  uncontrolled 
change 

EXPERIENCE 

Extensive  application 

Some  dependency  on 
new  technology 

Incompatible  with 
existing  technology 

REQUIREMENTS 

DEFINITION 

Known,  baselined 

Baselined,  some 
unknowns 

Unknown,  no 
baseline 

STABILITY 

Little  or  no  change 
projected 

Controllable  change 
projected 

Rapid  or  uncontrollable 
change 

COMPLEXITY 

Compatible  with 
existing  technology 

Some  dependency  on 
new  technology 

Incompatible  with 
existing  technology 

IMPACT 

Realistic,  achievable 
schedule 

Possible  slippage 
in  IOC 

Unachievable 

IOC 
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Accident  Risk,  FW-1 

Acquisition  Plan  (AP),  4-5,  5-11, 

5-56  thru  5-59 

Acquisition  Strategy, 

See  Also ,  Procurement  Strategy 
See  Also ,  DoD  Acquisition  Process 
5-7,  5-56  thru  5-59,  6-3  thru  6-11, 

7-1,  A--12 
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See  Arrow  Diagram  Method 
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See  Precedence  Diagramming  Method 
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5-42  thru  5-44 

Air  Force  Systems  Command  (AFSC) 

Risk  Model,  5-38,  5-42 

Alarm  Value, 

See  Tolerance  Value 

Analogy  Comparisons,  4-5,  4-8, 

5- 1  thru  5-3,  5-7  thru  5-11,  6-12 

ARCS,  5-31 

Army  Streamlined  Acquisition  Process, 
C-9 
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Assumption,  4-11  thru  4-13,  5-2  thru  5-3, 

6- 5 
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Model  (APPDM),  5-17 
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6-17  thru  6-18 

Avoidance,  4-10  thru  4-11,  5-2  thru  5-3, 
5-12,  5-18  thru  5-19,  6-5 
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Barcharts,  5-32 

Baseline  Cost  Estimate,  5-26  thru  5-28, 

5- 42  thru  5-44 

Baseline  Risk,  4-5  thru  4-6,  5-42  thru  5-44. 

6- 4 

Betting  Technique,  F-3  thru  F-ll 

C 

Carlucci  III,  Frank  C.  -  Deputy  Secretary 
of  Defense,  2-1 

Carlucci  initiatives,  2-1 

Coefficient  of  Variation,  5-32 
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Color  Coding,  5-48 

Committees  on  Armed  Services,  5-57 

Communication  Scheme,  5-3,  5-12,  5-15, 
6-1,  6-16,  6-18  thru  6-21,  8-2,  F-9  thru 
F-10 

Competition,  4-11,  5-59,  6-3 
Computer  Resources  Support,  3-7 
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Concept  Exploration,  5-59 

Concurrency,  4-12  thru  4-13,  5-29 

Confidence  Level,  5-33,  5-39,  5-41, 

E-4  thru  E-6 

Configuration  Control,  A-12 
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See  Also ,  Time  Constraints 
5-30  thru  5-34,  6-8  thru  6-10 

Contingency  Planning,  4-4,  4-13 

Contract  Costs,  5-26, 5-59 

Contract  (Type  of),  4-12,  6-3, 6-9 

Contract  Work  Breakdown  Structure  (CWBS), 
5-13  thru  5-14 

Contractor  Proficiency/Experience,  5-26,  7-4 

Contractor  Reviews,  4-11,  5-56 

Contractor  Support,  4-11,  6-18  thru  6-19, 
7-1 

Control,  4-11  thru  4-12,  5-2  thru  5-3, 

5-50,  6-12,  6-16,  7-3 

Cost  Analysis, 

See  Also ,  Cost  Estimating 
5-26,  5-34  thru  5-37,  5-41,  5-57 

Cost  Analysis  Strategy  Assessment  (CASA) 
Model,  5-37 

Cost  Analysis  Improvement  Group  (CAIG), 
5-57,  C-4 

Cost/Benefit  Analysis,  6-3,  6-9 

Cost  Breakdown  Structure,  5-43  thru  5-44 

Cost  Estimating, 

See  Also,  Independent  Cost  Estimating 
See  Also,  Estimating  Relationships 


4- 8,  5-10  thru  5-11,  5-26  thru  5-28, 

5- 33  thru  5-44,  5-57  thru  5-59,  6-3, 

6- 16  thru  6-17 

Cost  Estimating  Relationships  (CER), 

See  Also,  Estimating  Relationships 
5-26 

Cost  Growth,  5-55 

Cost  Performance  Reports  Analysis, 

5-3,  5-54  thru  5-58 

Cost  Plan,  4-12, 5-13,  5-31,  5-38 
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5- 48  thru  5-50,  5-54  thru  5-55, 6-2, 

6- 9  thru  6-16,  F-3,  F-6,  F-9  thru  F-10 

Cost  Risk/WBS  Simulation  Model,  5-3, 
5-37  thru  5-42,  6-11  thru  6-12,  6-16 

Critical  Activities, 

See  Also,  Critical  Path  Activities 
5-29, 5-32 

Critical  Path  Method  (CPM), 

See  Also,  Critical  Activities 
5-29,  5-32  thru  5-33,  E-10 

Criticality  Index,  5-32 

Cumulative  Density  Functions  (CDF), 
5-32,  E-2  thru  E-8 

Cumulative  Probability  Distribution, 

See  Also,  Probability  Distribution 
4-10,  5-37,  5-39,  6-20 
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Data  Collection  Process,  5-7  thru  5-10, 

5-28,  5-36  thru  5-39,  5-48,  5-56,  6-15 

Decision  Analysis,  5-2  thru  5-3, 

5- 21  thru  5-25,  6-4  thru  6-6, 

6- 11  thru  6-13,  C-7,  E-9  thru  E-10 

Decision  Table,  5-25 

Decision  Theory, 

See  Also ,  Decision  Analysis 

5- 22 

Decision  Tree,  5-22  thru  5-24,  E-2, 

E-9  thru  E-10 

Defense  System  Acquisition  Review  Council 
(DSARC),  C-5  thru  C-7 

Defense  Advisory  Board  (DAB),  5-57 

Defense  Acquisition  Board  (DAB)  Milestone 
Briefing,  5-44,  6-7,  6-9,  6-11 

Defense  Systems  Management  College 
(DSMC),  5-17,  5-34,  5-37,  6-7, 6-10, 

6- 17  thru  6-19,  G-2 

Degree  of  Risk, 

See  Level  of  Risk 

Degree  of  System  Definition,  5-26 

Delphi  Approach,  F-3,  F-9  thru  F-ll 

Demonstration/Validation  (DA/),  5-59, 7-1 

Dependency,  5-29  thru  5-30,  5-42, 

Deployment  Life  of  System  (DLS),  5-35,  A-l  1 

Design  Flexibility,  4-11 

Design  Guidance,  6-9,  6-11 

Design  Interface,  3-7 


Design  Review,  5-55 

Deterministic  Logic, 

See  Node  Logic 

Development  Costs,  5-35,  5-57 

Development  Test/Operational  Test,  C-10 

Development  Test  Quantities,  5-35 

Diagrammatic  Technique,  F-3 

Direct  Technique,  F-3  thru  F-4 

DoD  Acquisition  Policy  (on  Risk), 

Appendix  C 

DoD  Acquisition  Process, 

See  Also ,  System  Acquisition 

See  Also, DoD  Acquisition  Policy  (on  Risk) 

FW-1, 2-1, 2-3, 3-7,  4-3,  5-15,  5-37, 

A-8 

Downward  Allocation,  5-13 
Duration,  5-29, 6-9,  F-5  thru  F-6 

E 

Earned  Value  Performance  Measurement. 
5-50,  5-54 

Electronic  Systems  Division  (ESD)  Model, 
See  Estimating  Relationships 

Engineering  Change  Proposals  (ECP),  4-11 

5- 59 

Engineering  Complexity,  5-26 
Engineering  Development  Test,  C-10 
Estimating  Equation,  5-26  thru  5-28 

Estimating  Relationships,  5-3, 5-25  thru  5-28, 

6- 5  thru  6-6,  6-11  thru  6-14 

Evaluation  Criteria,  4-11 
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Expected  Monetary  Value  (EMV), 

5-22  thru  5-25,  E-2,  E-9  thru  E-10 

Experiment,  5-38 

Expert  Interviews/Judgment,  4-5,  4-8, 

5- 1  thru  5-7, 5-15, 5-28, 5-32, 5-43, 5-56, 

6- 4,  6-12  thru  6-13,  6-16,  8-2,  F-2  thru 
F— 11 

Expert  Support  Systems,  F-10 

Explanatory  Variables, 

See  Also ,  Independent  Variables 
5-26 

Extended  Contract  Work  Breakdown 
Structure  (Extended  CWBS), 

5-13  thru  5-14 

F 

Facilities,  3-7,  5-59,  A- 11 

Failure  Modes  Effects  and  Criticality  Analysis 
and  Repair  of  Repairables  Analysis,  A- 10 

Failure  Rate,  5-24,  5-35 

Finish  to  Finish  Relationship,  5-30 

Finish  to  Start  Relationship,  5-29  thru  5-30 

Fixed  Price  Contract,  4-11 

Frequency  Distribution,  5-38 

Functional  Analysis,  6  -2 

Full  Scale  Development  (FSD), 

5-17  thru  5-18,  5-59,  C-9  thru  C-10 

Fully  Integrated  Performance  Measurement, 
5-50  thru  5-54 


G 

General  Accounting  Office  (GAO)  Report, 
FW-1, 1-2, 2-1,  5-45 

Geometric  Mean,  5-27 

Government  Furnished  Properties  (GFP), 
5-54 

H 

High  Risk  Items,  4-11 

Histograms,  5-32 

Historic  Data,  5-8  thru  5-11,  5-27  thru  5-28, 
5-39,  6-14  thru  6-16 

I 

IBM  PCs,  5-42 

Independent  Cost  Analysis  (ICA),  5-57 

Independent  Cost  Estimate  (ICE),  5-3, 

5-57  thru  5-58,  C-14 

Independent  Technical  Assessment,  5-3, 

5-55  thru  5-56 

Independent  Variables, 

See  Also ,  Explanatory  Variables, 

5-26  thru  5-27 

Insurance  Risk,  FW-1 

Integrated  Logistics  Support  Plan  (ILSP), 

4-5,  5-11, 5-59,  A-8  thru  A-ll,  C-15 

Intermediate  Nodes,  5-31 

K 

Knowledge  and  Research,  4-11,  4-13, 

5-2  thru  5-3,  6-5 
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L 

Level  of  Risk,  2-3, 3-1  thru  3-2, 4-8  thru  4-12, 

5-4  thru  5-6,  5-39,  5-44,  6-17 

Lessons  Learned  Studies,  5-1  thru  5-2, 

5-7  thru  5-11,  5-19,  5-44,  6-12 

Life  Cycle  Cost  Model  (LCC),  3-12,  5-2  thru 

5- 3,  5-34  thru  5-37,  5-59,  6-3  thru  6-6, 

6- 11  thru  6-15,  A-9 

Line  Replaceable  Units  (LRU),  5-35 

Logistics  Reviews,  4-11,  6-3 

Logistics  Support  Analysis  (LSA),  4-11, 

5-34,  A-8  thru  A- 12 

Long  Lead  Items,  4-11,  5-59,  7-3 

Lower  Level  Risk  Matrix,  5-18 

M 

Maintenance  Planning,  3-7 

Management  Reserve  Funding,  5-25  thru 
5-28,  6-14 

Management  Risk,  5-13 

Manpower/Personnel,  3-7  thru  3-10,  5-18, 

5- 25  thru  5-26,  5-33,  5-36,  5-59,  6-1  thru 

6- 3,  6-6  thru  6-8,  6-21 

Manufacturing  Plan  (MP),  4-5,  5-11,  5-59, 

7- 2,  C-15 

Manufacturing  Process,  5-7,  5-11,  5-59 

Mathematical  Models,  3-2,  4-5,  4-8,  5-42, 

8- 1 

Maximize  Profit,  5-  24 


Mean 

See  Sample  Mean 

Measurement  Techniques,  4-4, 4-9,  5-15, 
5-45 

Minimize  Cost,  5-24 
Mode 

See  Sample  Mode 

Modified  Churchman/ Ackoff  Technique, 

F-3  thru  F-9,  F-ll 

Monte  Carlo  Simulation,  5-32  thru  5-34, 
5-38  thru  5-39,  6-20 

Multiple  Regression  Analysis,  5-27 

Multiple  Users,  5-26 

Multipliers,  5-42 

Mutually  Exclusive,  E-9 

N 

NAVAIR,  5-19 

Network  Analysis,  5-2  thru  5-3,  5-9,  5-22, 

5- 28  thru  5-34,  5-37,  5-54,  6-4  thru  6-7, 

6- 11  thru  6-13,  6-17,  6-20 

Network  Logic,  5-28  thru  5-35,  5-42 
Node  Logic, 

See  Also,  Network  Logic 
5-31 

Nodes, 

See  Also,  Intermediate  Nodes 
See  Also ,  Source  Nodes 
See  Also,  Terminal  Nodes 
5-31 

Numbering  Systems/Schemes,  5-13 
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o 

Office  of  the  Secretary  of  Defense  (OSD),  5-57 
OPERA,  5-33 

Operating  Cost,  5-35,  5-59, 6-15 

Operational  Risk,  3-12,  4-3,  5-37, 

5-59,  7-3 

Opportunity,  3-2  thru  3-3 

P 

Packaging,  Handling,  Storage,  and 
Transporation  (PHS&T),  3-7 

Packard,  David  -  Deputy  Secretary 
of  Defense,  2-1 

Parametric  Cost  Estimating, 

See  Also ,  Cost  Estimating  Relationships 
5-26  thru  5-27,  5-44,  6-13  thru  6-14,  C-5 

Payoff,  5-22  thru  5-23,  7-2,  E-9 

Percentage  Bands,  5-48 

Performance  Incentives,  4-12 

Performance  Parameters, 

See  Technical  Performance  Requirements 

Performance  Risk,  2-3,  3-4  thru  3-12, 

4- 9  thru  4-10,  5-3  thru  5-8,  5-13, 

5- 17  thru  5-18, 5-29, 5-34, 5-45, 5-48  thru 
5-50,  6-9,  F-2  thru  F-ll 

Performance  Tracking,  5-2  thru  5-3,  5-13, 
5-29, 5-45  thru  5-54, 6-11  thru  6-12, 6-17 

Physical  Control  Space  (PCS),  7-3 

Planned  Utilization  Rate,  5-35 


Planning,  Programming,  and  Budgeting 
System  (PPBS)  Process,  C-2 

Point  Cost  Estimate  (PCE),  5-39  thru  5-43 

Polaris  Submarine  Program,  5-29 

POM/BES  Preparation,  5-43  thru  5-44, 
6-14  thru  6-16 

POM/Budget  Submittal,  6-9  thru  6-14 

Post  Production  Support  (PPS)  Planning, 
A- 11  thru  A- 12 

Precedence  Diagramming  Method  (PDM), 
5-30 

Predecessor/Successor  Activities, 

5-29  thru  5-30 

Prediction  Equation, 

See  Estimating  Equation 

Pre-Planned  Product  Improvements  (P3I). 

4- 11,  C-10 

Probabilistic  Branching,  5-22  thru  5-23 

Probabilistic  Logic 
See  Node  Logic 

Probability  Curves, 

See  Also,  Probability  Density  Functions 
See  Also,  Cumulative  Density  Functions 

5- 32,  5-39 

Probability  Density  Function  (PDF), 

5-5  thru  5-7,  5-32  thru  5-34,  6-13, 

E-2  thru  E-8,  F-2 

Probability  Distributions, 

See  Also,  Cumulative  Probability 
Distribution 

See  Also,  Probability  Density  Function 
3-2,  5-38 

Probability  Estimates,  5-22  thru  5-25 
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Probability  Functions, 

See  Also,  Probability  Density  Function 
See  Also,  Cumulative  Density  Function 
5-31 

Probability  of  Occurrence  (of  the  Event), 

3- 1  thru  3-2, 3-13,  4-6  thru  4-8,  5-6, 
5-22  thru  5-23,  7-4,  E-9,  F-6  thru  F-8 

Probability  Theory,  3-1  thru  3-2,  4-5,  5-31, 
Appendix  E,  Appendix  F 

Problem  Definition,  5-22  thru  5-24 

Procedures,  6-21  thru  6-22 

Procurement  Document  Generator  (PDG), 

5- 17 

Procurement  Strategy,  5-13,  7-1 
Production  Cost,  4-11,  5-35,  5-37,  5-59,  7-2 

Production  Phase,  5-18, 5-35, 5-57  thru  5-59, 

6- 4 

Production  Rate  and  Quantity  Analysis 
(PRQA),  5-34  thru  5-36,  7-2 

Production  Readiness  Reviews  (PRR), 

4- 11,  5-59,  C-5 

Production  Schedule,  4-11,  5-34,  5-59 

Program  Advocacy,  3-10,  5-57  thru  5-58, 
6-19 

Program/Contract  Description  Characteristic 
Data,  5-25  thru  5-28 

Program  Evaluation  and  Review  Technique 
(PERT),  5-29  thru  5-30,  E-10 

Program  Goals,  4-6  thru  4-8, 5-17, 5-48, 6-1 

Program  Management  Directive,  4-6,  5-50 

Program  Management  Plan  (PMP), 

4-3  thru  4-4,  5-11,  5-28 


Program  Office  Cost  Estimate  (PCE), 

5-57  thru  5-58 

Program  Plan, 

See  Also,  Program  Plan  Evaluation 

4- 5, 5-17  thru  5-18,  5-33,  5-43  thru  5-44, 

5- 50,  5-56 

Program  Plan  Evaluation,  5-1  thru  5-3, 

5-11  thru  5-19,  6-12 

Program/Project  Management 
See  Project/Program  Management 

Program  Schedule,  4-3  thru  4-5,  5-28  thru 
5-32, 5-51  thru  5-54, 7-1  thru  7-3,  F-3  thru 
F-6,  F-9  thru  F-10 

Program  Scope,  5-29,  6-13 

Program  Status  Reporting,  6-9  thru  6-11 

Program  Strategy,  4-6  thru  4-8,  5-17 

Program  Status  Reporting, 

See  Tracking 

Program  Unique  Technical  Indicator 
5-45 

Programmatic,  3-4,  3-6  thru  3-9,  3-13,  4-4, 
5-17  thru  5-21,  5-32,  6-1,  6-10 

Programmatic  Risk  Sources,  A-4  thru  A-7 

Project/2,  5-33 

Project/Program  Management, 

FW-1, 2-3,  3-1,  3-9  thru  3-10,  4-3  thru 

4- 6,  4-12  thru  4-13,  5-1,  5-17  thru  5-19, 

5- 32 

Project  Summary  Work  Breakdown  Structure 
(PSWBS),  5-13  thru  5-14 

Project  Work  Breakdown  Structure  (PBWS), 
5-12  thru  5-14 

PROSIM,  5-33 
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Q 

Quick  Cost  Model,  5-34 

Quick  Reaction  Cost  Model, 

See  Also,  Life  Cycle  Cost  Models 

5-34 

Quick  Reaction  Models, 

See  Also,  Life  Cycle  Cost  Models 

4-10  thru  4-13,  5-2,  5-34 

R 

Rand  Corporation,  F-9 
Random  Number  Generator,  5-38 
Range  of  Uncertainty,  5-5 

Rating  of  Risk,  3-2,  3-13,  4-4,  4-8, 

4- 13, 5-1,  5-13,  5-21 

Rating  Scheme, 

See  Rating  of  Risk 

Regression  Analysis,  5-26  thru  5-27 

Reliability  and  Maintainability  (R&M),  7-1 
thru  7-2,  A-8  thru  A-9,  F-ll 

Repair  Level  Analysis,  5-36 

Request  for  Proposal  (RFP),  5-19,  7-1,  7-4 

Resource  Requirements,  5-29, 5-36  thru  5-37, 

5- 41  thru  5-44,  5-50,  5-56  thru  5-59,  6-1 
thru  6-20,  7-3,  F-3,  F-10  thru  F-ll 

Risk  Analysis  and  Management  Survey,  6-1 

Risk  Analysis, 

See  Also ,  Risk  Analysis  Techniques 
See  Also,  DoD  Acquisition  Policy 
2-2, 3-10, 4-1  thru  4-13, 5-2  thru  5-3, 5-23 


thru  5-25, 5-32  thru  5-37,  5-43, 5-55  thru 

5-58, 6-1  thru  6-20, 7-1  thru  7-4, 8-1  thru 
8-2 

Risk  Analysis  Techniques, 

See  Also,  Risk  Analysis 

5- 4  thru  5-9,  5-29  thru  5-33,  5-38,  5-43, 

6- 1,  6-5  thru  6-19 

Risk  Assessment, 

See  Also,  Risk  Analysis 
1-2, 2-1  thru  2-2, 4-1  thru  4-13,  5-1  thru 
5-10, 5-22, 5-33  thru  5-34, 5-50, 5-55  thru 
5-59,  6-2  thru  6-10,  6-15,  7-1,  7-4,  E-4, 
F-2  thru  F-ll 

Risk  Assessment  Report 
See  Also,  Risk  Assessment 

7- 1 

Risk  Averse,  FW--3,  8-1,  E-9,  F-3 
Risk  Budgeting 

See  Also,  Risk  Funds  Budgeting 

4- 4,  *5  —41  thru  5-44,  6-5 

Risk  Classification 
See  Risk  Facets 

Risk  Drivers 
See  Also,  Risk  Indicators 
3-8  thru  3-9 

Risk  Exposure,  4-12 

Risk  Facets,  FW-3,  3-4  thru  3-9,  3-13, 

5- 4,  5-21,  5-58,  6-5,  7-3 

Risk  Factors,  5-3,  5-13,  5-42  thru  5-45, 

6- 5  thru  6-6,  6-11  thru  6-12,  6-16  thru 

6-17 

Risk  Funds  Budgeting,  5-25  thru  5-28, 

5-35  thru  5-39,  6-14  thru  6-17 

Risk  Handling, 

See  Also,  Risk  Handling  Techniques 
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See  Also,  Risk  Analysis 

2- 2,  3-10,  4-2  thru  4-4,  4-10  thru  4-13, 
5-2  thru  5-3,  5-33,  5-58,  6-1  thru  6-10, 
8-2,  G-2 

Risk  Handling  Techniques, 

See  Also,  Risk  Analysis 

5- 3,  5-8  thru  5-9,  5-58  thru  5-59,  6-1, 

6- 5,  6-18  thru  6-19,  G-2 

Risk  Identification,  2-3,  4-4  thru  4-9, 

4- 13,  5-1  thru  5-9,  5-13  thru  5-21,  6-3 
thru  6-5,  7-4,  8-1  thru  8-2 

Risk  Indicators, 

See  Also,  Risk  Drivers 

3- 8  thru  3-9 

Risk  Management  Plan,  4-3  thru  4-5, 

5- 17,  7-1 

Risk  Quantification 

4- 5  thru  4-8,  4-13,  5-1  thru  5-6, 

6- 4  thru  6-5,  7-4 

Risk  Planning, 

See  Also,  Risk  Analysis 

2-2,  4-1  thru  4-5,  4-13,  5-2,  5-12,  5-18, 

5- 58,  6-1  thru  6-16 

Risk  Prone 
See  Risk  Taker 

Risk  Reduction  Plan,  4-2  thru  4-6,  4-12, 

5- 18  thru  5-19,  5-55  thru  5-58,  6-3  thru 

6- 5,  6-16,  7-1  thru  7-4 

Risk  Sources, 

See  Also,  Technical  Risk  Sources 
FW-3,  1-1,  3-4  thru  3-8,  4-10,  5-33  thru 
5-34,  5-58 

Risk  Takers,  FW-3,  4-8,  8-1 


Risk  Techniques, 

See  Also,  Risk  Analysis  Techniques 
See  Also ,  Risk  Handling  Techniques 

6-1  thru  6-22 

RISNET,  5-33 

S 

Safety  Risk,  FW-1, 2-2  thru  2-3 

Sample  Mean,  5-32,  E-4  thru  E-9 

Sample  Mode,  5-32,  E-4  thru  E-9 

Schedule  Assessment,  5-9,  5-29,  6-3 

Schedule  Dictionary,  5-15 

Schedule  Performance  Index,  5-53  thru  5-54 

Schedule  Plan, 

See  Also .  Program  Schedule 
4-12,  5-9,  5-13,5-29,  5-51 

Schedule  Risk,  2-3,  3-4  thru  3-13, 

4- 4  thru  4-5,  4-9  thru  4-10,  5-3  thru 

5- 10,  5-18,  5-28  thru  5-34,  5-51  thru 
5-54,  6-2,  6-9  thru  6-10 

Schedule  Slippage,  5-34,  5-54,  A-ll 

Scheduled  Completion  Date,  5-29  thru  5-34 

Science  and  Technology  Objectives  Guide 
(STOG),  C-10 

Security,  7-3 

Sensitivity  Analysis, 

See  Also,  “What-If  ’  Analysis 
4-9,  5-28,  5-34  thru  5-35 

Service  Requirements,  C-6 

Severity  of  Consequence/Impact,  3-2,  3-13, 
4-8,  5-3  thru  5-6 
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Simulation, 

See  Also,  Monte  Carlo  Simulation 

See  Also,  Cost/Risk  WBS  Simulation  Model 

5- 2,  5-31,  5-37,  6-5 

Slack  Time,  5-30 

Software  Quality  Assurance  Program,  G-2 
Software  Reporting  Metrics,  G-2  thru  G-7 
Source  Nodes,  5-31 

Source  Selection,  4-11,  6-6  thru  6-11, 

6- 15  thru  6-20,  7-4 

Special  Technical  Indicators,  5-48  thru  5-49 

Specifications/Specification  Tree,  5-12  thru 

5- 18,  5-51  thru  5-59 

Standard  Deviation,  5-32,  E-4  thru  E-6 

Standard  Technical  Indicators,  5-45  thru  5-50, 

6- 17  thru  6-18 

Start-to-Finish,  5-30 
Start-to-Start,  5-30 

Statements  of  Work  (SOW),  5-12,  5-15  thru 
5-18 

States  of  Nature,  5-21  thru  5-25 
Statistical  Independence,  E-9  thru  E-10 
Success  Criteria,  2-2 

Summary  Work  Breakdown  Structure  (SWBS), 
5-13  thru  5-14 

Support  Cost,  4-11,  5-35,  6-15 
Support  Equipment,  5-35  thru  5-36 
Supportability, 

See  Also,  Supportability  Risk  Sources 

3-4  thru  3-13,  4-4,  5-17  thru  5-18,  5-52, 
5-59,  6-2,  6-10,  7-1 


Supportability  Risk  Sources,  A-7  thru  A- 12 

Suspect  Basis,  5-9 

System  Acquisition, 

See  Also,  DoD  Acquisition  Process 
See  Also,  Acquisition  Strategy 
FW-1, 2-1, 3-6  thru  3-9,  4-1,  4-8  thru 

4- 12,  5-8  thru  5-11,  5-19,  5-26,  5-57  thru 

5- 58 

Systems  Engineering  Management  Plan 
(SEMP),  4-4,  5-11 

T 

Technical  Advancement  Progress, 

See  Performance  Tracking 

Technical  Complexity,  5-8  thru  5-11,  5-17 

Technical  Data,  3-7,  4-11 

Technical  Indicators, 

See  Standard  Technical  Indicators 

Technical  Performance  Plan, 

See  Also,  Technical  Performance 
Requirements 

4- 12,  5-13,  5-51 

Technical  Performance  Requirements, 

5- 13  thru  5-17, 5-28, 5-31  thru  5-34, 5-45, 
5-50  thru  5-54,  5-58,  F-2  thru  F-6 

Technical  Risk, 

See  Also,  Technical  Risk  Dictionary 
See  Also,  DoD  Acquisition  Policy 
1-2,  2-1,  3-4  thru  3-13,  4-4,  5-10  thru 

5- 21,  5-31,  5-42  thru  5-50,  5-58,  6-2, 

6- 9  thru  6-15,  7-2  thru  7-3 

Technical  Risk  Assessment  Report, 

5-45, 5-50 
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Technical  Risk  Dictionary,  5-15  thru  5-18 

Technical  Risk  Sources,  A-2  thru  A-4 

Technology  Assessment, 

See  Also,  Risk  Assessment 
5-2,  5-11 

Terminal  Nodes,  5-31 

Test  and  Evaluation  Master  Plan  (TEMP) 

4-3  thru  4-5,  5-11,  5-17, 5-59, 7-2,  A-10, 
C-4,  C-10  thru  C-16 

The  Analytic  Sciences  Corporation  (TASC), 
FW-3 

Time  Constraints,  5-31  thru  5-32,  F-3 
Tolerance  Bands,  5-48 
Tolerance  Value,  5-51 
Tooling  Costs,  5-35 

Top  Level  Risk  Matrix,  4-6.  5-17  thru  5-18 
Total  Failure,  4-5 

Total  Risk  Assessing  Cost  Estimate  (TRACE), 
C-ll  thru  C-13 

Total  Success,  4-5 

Tracking, 

See  Also,  Performance  Tracking 

4- 9  thru  4-12,  5-21,  5-29,  5-44,  5-48  thru 

5- 50,  6-16  thru  6-17 

Tradeoff  Analysis,  5-34  thru  5-36, 6-3  thru 

6- 10,  7-1 

Training,  3-7,  6-21  thru  6-22 

Training  Support 
See  Also  Training 

Transfer,  4-11  thru  4-13,  5-2  thru  5-3,  6-5 

Transition  Templates,  5-1  thru  5-3,  5-19  thru 
5-21,  6-11  thru  6-17 


U 

Undersecretary  of  Defense  Research  and 
Engineering  (USDR&E),  5-57 

Upward  Summarization,  5-13 

Uncertainty, 

See  Also ,  Range  of  Uncertainty 

3- 1,  4-5,  5-21  thru  5-31,  5-38  thru  5-41. 

6-9  thru  6-16,  C-4,  F-2  thru  F-3 

USAF  Electronic  Systems  Division, 

5-27  thru  5-28 

U.S.  Army  TRACE  Cost  Risk  Procedures, 
5-44 

V 

Variance, 

See  Standard  Decoration 
Verification/Validation, G-2 
VERT/VERT  PC,  5-33,  5-37 

W 

Warranty,  4-11  thru  4-13,  5-34  thru  5-36, 
5-59,  6-5,  6-9 

Watchlist,  4-9  thru  4-11,  5-2  thru  5-9, 

5-18  thru  5-21,  5-44,  6-4 

Weighted  Average,  5-27 

“What-If ’  Analysis, 

See  Also,  Sensitivity  Analysis 

4- 9, 5-3,  5-29 

Work  Breakdown  Structure  (WBS), 

See  Also,  Work  Breakdown  Structure 
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Index/Dictionary 

See  Also,  Protect  Work  Breakdown 
Structure 

5-1  thru  5-2,  5-12  thru  5-18,  5-29, 

5-37  thru  5-45, 5-51, 6-16  thru  6-17,  C-ll 

Work  Breakdown  Structure  (WBS) 
Index/Dictionary,  5-12  thru  5-15 

Z 

Zenith  100  Computer,  5-42 
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by  all  levels  of  program  managers  and  designated  "risk"  analysts,  f  e.yu  - 
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